Reassignment of a virtual connection from a busiest virtual connection or locality domain to a least busy virtual connection or locality domain

ABSTRACT

A method and system for load balancing for virtual connections over a fiber channel network, the virtual connections using SCSI. The method including finding a one-way reassignment of a virtual connection (VC) from busiest VCE or locality domain to least busy VCE or locality domain placing the busiest VCE or locality domain and least busy VCE or locality domain in a target load range without reversing a load order, and executing the reassignment by a server fiber channel adapter.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to a co-pending application of patentapplication Ser. No. 13/725,652, filed Dec. 21, 2012. This applicationis related to a co-pending application of patent application Ser. No.13/725,668, filed Dec. 21, 2012. This application is related to aco-pending application of patent application Ser. No. 13/725,696, filedDec. 21, 2012. This application is related to a co-pending applicationof patent application Ser. No. 13/725,816, filed Dec. 21, 2012. Thisapplication is related to a co-pending application of patent applicationSer. No. 13/725,823, filed Dec. 21, 2012. This application is related toa co-pending application of patent application Ser. No. 13/725,845,filed Dec. 21, 2012. This application is related to a co-pendingapplication of patent application Ser. No. 13/725,850, filed Dec. 21,2012. This application is related to a co-pending application of patentapplication Ser. No. 13/725,726, filed Dec. 21, 2012. This applicationis related to a co-pending application of patent application Ser. No.13/725,737, filed Dec. 21, 2012. This application is related to aco-pending application of patent application Ser. No. 13/725,748, filedDec. 21, 2012. This application is related to a co-pending applicationof patent application Ser. No. 13/725,765, filed Dec. 21, 2012. Thisapplication is related to a co-pending application of patent applicationSer. No. 13/725,860, filed Dec. 21, 2012. This application is related toa co-pending application of patent application Ser. No. 13/725,819,filed Dec. 21, 2012.

FIELD OF INVENTION

Embodiments of the present invention relate generally to data storagesystems. More particularly, embodiments of the invention relate to datacommunicated across a Fibre Channel network.

BACKGROUND

In modern computer systems, a file system stores and organizes computerfiles to enable a user to efficiently locate and access requested files.File systems can utilize a storage device such as a hard disk drive toprovide local access or provide access to data stored on a remote fileserver. A file system can also be characterized as a set of abstractdata types that are implemented for the storage, hierarchicalorganization, manipulation, navigation, access, and retrieval of data.The file system software is responsible for organizing files anddirectories.

Many companies and individuals with large amounts of stored data employa file system as a data storage system. These data storage systems canbe located local to the data to be backed up or at a remote site. Thedata storage systems can be managed by the entity controlling the datastorage devices or a data storage service company. Data can be added tothe storage system at any frequency and at any amount.

Data storage systems may offer storage for backup and disaster recovery.Transfer to remote storage may require the transfer of data over anetwork. One network that allows transferring data across a data storagesystem is a Fibre Channel network. Fibre Channel allows a server and/ora storage unit to be located at a substantial distance from othercomponents of the data storage system if optical fiber is used as thephysical medium. However, optical fiber is not required for shorterdistances, as a Fibre Channel network may also be implemented usingcoaxial cable and ordinary telephone twisted pair.

SUMMARY

A method is implemented by a server for load balancing of virtualconnections over a fiber channel network. The virtual connections useSmall Computer System Interface (SCSI). The method includes finding aone-way reassignment of a virtual connection (VC) from busiest virtualconnection engine (VCE) or locality domain within the server to leastbusy VCE or locality domain within the server that places the busiestVCE or locality domain and least busy VCE or locality domain in a targetload range without reversing a load order and executing the reassignmentby a server fiber channel adapter.

A server system implements load balancing for virtual connections over afibre channel network. The virtual connections use Small Computer SystemInterface (SCSI). The server system includes a host adapter to transmitand receive data traffic over a fibre channel connection. The serversystem further includes a processor to execute a server fibre channeladapter. The server fibre channel adapter includes a load balancingengine to manage load across a set of locality domains and virtualconnection engines. The load balancing engine is configured to find aone-way reassignment of a virtual connection (VC) or from busiestvirtual connection engine (VCE) or locality domain within the serverfibre channel adapter to least busy VCE or locality domain within theserver fibre channel adapter that places the busiest VCE or localitydomain and least busy VCE or locality domain in a target load rangewithout reversing a load order and to execute the reassignment.

A non-transitory machine readable medium has stored therein instructionsto be executed by a server computer. The instructions when executed bythe server computer cause the server computer to find a one-wayreassignment of a virtual connection (VC) from busiest virtualconnection engine (VCE) or locality domain within the server computer toleast busy VCE or locality domain placing within the server computerthat places the busiest VCE or locality domain and least busy VCE orlocality domain in a target load range without reversing a load orderand to execute the reassignment.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments of the invention are illustrated by way of example andnot by way of limitation in the figures of the accompanying drawings inwhich like references indicate similar elements. It should be noted thatreferences to “an” or “one” embodiment of the invention in thisdisclosure are not necessarily to the same embodiment, and they mean atleast one.

FIG. 1 is a block diagram of one embodiment of a data storage system.

FIG. 2 is a block diagram of one embodiment of a client of a datastorage system.

FIG. 3 is a block diagram of one embodiment of a server of a datastorage system.

FIG. 4A is a conceptual block diagram illustrating communication pathsover a Fibre Channel network connecting a client with a server accordingto one embodiment of the invention.

FIG. 4B is a block diagram illustrating one embodiment of a SCSI requestadapted for communication over a Fibre Channel network from a client toa server.

FIG. 4C is a block diagram illustrating one embodiment of a SCSIresponse adapted for communication over a Fibre Channel network from aserver to a client.

FIG. 4D is a block diagram illustrating one embodiment of a logicalblock address field included in a command descriptor block of a SCSIrequest adapted for communication over a Fibre Channel network from aclient to a server.

FIG. 5 is a flowchart illustrating one embodiment of a method forinitializing a client that is connected with a server by a Fibre Channelnetwork.

FIG. 6 is a flowchart illustrating one embodiment of a method executedby a client for establishing a virtual connection with a server over aFibre Channel network.

FIG. 7 is a flowchart illustrating one embodiment of a method executedby a client for communicating with a server over a Fibre Channelnetwork.

FIG. 8 is a flowchart illustrating one embodiment of a method forinitializing a server that is connected with a client by a Fibre Channelnetwork.

FIG. 9 is a flowchart illustrating one embodiment of a method executedby a server for a server messaging service.

FIG. 10 is a flowchart illustrating one embodiment of a method executedby a server for establishing a virtual connection with a client over aFibre Channel network.

FIG. 11 is a flowchart illustrating one embodiment of a method executedby a server for communicating with a client over a Fibre Channelnetwork.

FIG. 12 is a flowchart illustrating one embodiment of a method executedby a server for a server messaging service.

FIG. 13 is a flowchart illustrating one embodiment of a method executedby a client for reliably communicating with a server over a FibreChannel network.

FIG. 14 is a flowchart illustrating one embodiment of a method executedby a server for reliably communicating with a client over a FibreChannel network.

FIG. 15 is a flowchart illustrating one embodiment of a method executedby a server for selecting paths for virtual connections.

FIG. 16 is a flowchart illustrating one embodiment of a method executedby a server for rebalancing virtual connections over available paths.

FIG. 17 is a block diagram of one embodiment of a client-server systemfor reliable communication over a Fibre Channel network.

FIG. 18 is a flowchart illustrating one embodiment of virtual connectionengine instantiation.

FIG. 19 is a flowchart illustrating one embodiment of virtual connectiongeneration and load distribution.

FIG. 20 is a block diagram of one embodiment of a client-server systemfor reliable communication over a Fibre Channel network.

FIG. 21 is a flowchart illustrating one embodiment of a virtualconnection rebalancing process.

FIG. 22 is a block diagram of one embodiment of shared access system formanaging data streams in virtual connections.

FIG. 23 is a flowchart illustrating one embodiment of a consumer methodfor shared data stream management in a virtual connection.

FIG. 24 is a flowchart illustrating one embodiment of a producer methodfor shared data stream management in a virtual connection.

FIG. 25 is a block diagram of one embodiment of a statistics managementmodule of a server Fibre Channel adapter.

FIG. 26 is a flowchart illustrating one embodiment of a statisticalmonitoring process.

FIG. 27 is a flowchart illustrating one embodiment of a statisticalmonitoring process having a set of specified cases for generatingmonitoring data for a given interval.

FIG. 28 is a block diagram of one embodiment of a VCE load balancingengine.

FIG. 29 is a flowchart illustrating one embodiment of a method of VCErebalancing.

FIG. 30 is a flowchart illustrating one embodiment of a method ofendpoint assignment.

FIG. 31 is a flowchart illustrating one embodiment of a method ofendpoint rebalancing.

DETAILED DESCRIPTION

Several embodiments of the invention with reference to the appendeddrawings are now explained. The following description and drawings areillustrative of the invention and are not to be construed as limitingthe invention. Numerous specific details are described to provide athorough understanding of various embodiments of the present invention.However, in certain instances, well-known or conventional details arenot described in order to provide a concise discussion of embodiments ofthe present inventions.

Reference in the Specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin conjunction with the embodiment can be included in at least oneembodiment of the invention. The appearances of the phrase “in oneembodiment” in various places in the Specification do not necessarilyall refer to the same embodiment.

FIG. 1 is a block diagram illustrating a data storage system 100according to one embodiment of the invention. The data storage system100 includes, but is not limited to, one or more client systems 110a-110 b communicatively coupled by a Fibre Channel (FC) network 130 witha server 150 connected with one or more storage units 180 a-180 b.

To efficiently transfer data within a data storage system, a request ina data storage system can be sent using a Small Computer SystemInterface (SCSI) request. SCSI requests traditionally specify a logicalblock address to be written to or to be read. These requests may be sentover a Fibre Channel network by packaging the SCSI requests as FibreChannel frames, and unpackaging the SCSI request at the recipient.Responses to SCSI requests may be likewise received over the FibreChannel network.

A client 110 can be any type of client such as a personal computer(e.g., desktops, laptops, and tablets), a workstation, a handhelddevice, a Web-enabled appliance, a gaming device, a media player, or amobile phone (e.g., Smartphone), or any computing system operable tocommunicate over a Fibre Channel network.

SCSI requests are sent from by clients 110 a-110 b and received at theserver 150 across the FC network 130. FC network 130 can be any type ofnetwork using Fibre Channel. In one embodiment, the FC network 130 is astorage area network (SAN). The FC network 130 can feature any suitablenetwork topology. Thus, the FC network 130 can be a point-to-pointnetwork. Alternatively, the FC network 130 can be an arbitrated loopnetwork. In another embodiment, the FC network 130 can be a switchedfabric network. In such embodiments, the FC network 130 can include oneor more Fibre Channel switches (not shown) and visibility of the server150 and/or clients 110 a-110 b can be controlled with Fibre Channelzoning.

The server 150 can include any type of server or cluster of servers. Forexample, the server 150 can be a storage server used for any of variousdifferent purposes, such as to provide multiple users with access toshared data and/or to back up mission-critical data. The server 150 canbe, for example, a file server (e.g., an appliance used to provide NAScapability), a block-based storage server (e.g., used to provide SANcapability), a unified storage device (e.g., one which combines NAS andSAN capabilities), a nearline storage device, a direct attached storage(DAS) device, a tape or virtual tap backup device, or essentially anyother type of data storage device or a combination thereof. The server150 can have a distributed architecture, or all of its components can beintegrated into a single unit. The server 150 can be implemented as partof an archive and/or backup system such as a deduplication storagesystem available from EMC® Corporation of Hopkinton, Mass. Additionally,the server 150 can be communicatively coupled with an auxiliary storagesystem (not shown) similar to the server 150. The auxiliary storagesystem can duplicate the functionality of the server 150. Alternativelyor in addition to the server 150, the auxiliary storage system canprovide some additional data warehousing or data manipulation.

As shown in FIG. 1, the server 150 is coupled with one or more storageunits 180 a-180 b. A storage unit 180 can be implemented locally (e.g.,single-node operating environment) or remotely (e.g., multi-nodeoperating environment) via an interconnect 170, which can be a bus or anetwork. In one embodiment, one of the storage units 180 a-180 boperates as an active storage unit to receive and store external orfresh data, while the other storage unit operates to periodicallyarchive data from the active storage unit according to an archivingpolicy or scheme. A storage unit 180 can be, for example, conventionalmagnetic disks, optical disks such as CD-ROM or DVD based storage,magnetic tape storage, magneto-optical (MO) storage media, solid statedisks, flash memory based devices, or any other type of non-volatilestorage devices suitable for storing large volumes of data. The storageunits 180 a-180 b can also be combinations of such storage devices. Insome embodiments, the storage units 180 a-180 b can be organized intoone or more volumes of Redundant Array of Inexpensive Disks (RAID).

A simple embodiment of a client 200 is illustrated at FIG. 2. The client200 can be or can include one of clients 110 a-110 b of FIG. 1. In oneembodiment, the client 200 includes, but is not limited to, severalcomponents: including a user interface 220, main memory 215, a clienthost bus adapter 216, storage 217, and a processor 218. These componentscan be communicatively coupled via a bus 219. The bus 219 can be anycommunication subsystem or medium adapted to transfer data within theclient 200. The bus 219 can be a plurality of computer buses and includeadditional circuitry to transfer data.

The user interface 220 can allow a user to interact with the client 200,such as a through a graphical user interface (GUI) provided by a module212-213 or through a command line interface. To realize this, the client200 can include or can be communicatively coupled with one or morehardware devices (not shown), such as a display and one or more devicessuitable for user input (e.g., a keyboard, a mouse, or touch screen).

Storage 217 can be implemented locally (e.g., single-node operatingenvironment) via bus 219 (as shown) or remotely (e.g., multi-nodeoperating environment) via a network (not shown). Storage 217 can be,for example, conventional magnetic disks, optical disks such as CD-ROMor DVD based storage, magnetic tape storage, magneto-optical (MO)storage media, solid state disks, flash memory based devices, or anyother type of storage devices suitable for storing data. In someembodiments, storage 217 includes registers, caches or other similartemporary memory components. Though illustrated as a single device,storage 217 can be a combination of several devices, such as volatileand non-volatile memory devices.

The processor 218 can be any processor suitable to execute instructionsof the components 211-214 stored in main memory 215. Accordingly, theprocessor 218 can be, for example, a central processing unit (CPU), amicroprocessor, a network processor or other similar processing device.In some embodiments, the processor 218 includes a plurality ofprocessors, such as a dedicated processor (e.g., a graphics processingunit), a network processor, a front end processor, or any processorsuitable to execute operations of the client 200 connected with a serverby a Fibre Channel network.

Main memory 215 may be coupled with the processor 218. In someembodiments, main memory 215 provides storage of computer readableinstructions, data structures, program and application modules, andother data for the client 200. Main memory 215 can include, but is notlimited to, a client operating system (OS) Small Computer SystemInterface (SCSI) service 211, a data optimization module 212, a datastorage module 213, and a Fibre Channel (FC) transport adapter 214.

The client OS SCSI service 211 is operable to discover SCSI devices,send SCSI requests and receive SCSI responses across a FC network usinga client host bus adapter (HBA) 216. The client OS SCSI service 211 caninclude any SCSI interface, such as the Windows SCSI Pass ThroughInterface (SPTI) or the Linux SCSI subsystem. In some embodiments, theclient OS SCSI service 211 is operable to discover SCSI devices as oneor more logical unit numbers (LUN) advertised by a server. The client OSSCSI service 211 can discover a LUN when the client OS SCSI service 211is loaded or at any point thereafter—e.g., the client OS SCSI service211 can be configured to discover the available LUNs at boot time, toperiodically discover new or removed LUNs, or to discover available LUNsin the event of an error in communicating with a previously discoveredLUN. The client OS SCSI service 211 can create one or more SCSI deviceentries, such as in a device directory, for each discovered LUN. In someembodiments, multiple SCSI device entries are created at the client 200for a single LUN to indicate that a client can access that LUN overmultiple paths (e.g., one LUN may be advertised at multiple ports of theserver host bus adapter 330 at the server 300, and visible throughmultiple ports of the client host bus adapter 216). These client-sideSCSI device entries can be accessed by other components of the client200, such as the FC transport adapter 214. In some embodiments, theclient 200 can support multi-pathing and therefore a single SCSI deviceentry is created.

The client OS SCSI service 211 can scan the available SCSI devices bysending a SCSI inquiry request for each LUN. The client OS SCSI service211 can receive a SCSI inquiry response that includes inquiryinformation associated with the advertised LUN. In some embodiments, theinquiry information includes an indication that this LUN can receiveSCSI requests from the client 200 over a FC network, such as a field ofthe SCSI inquiry response that contains a specific value. A LUN havingsuch an indication represents a transport path between the client 200and a server over the FC network (e.g., a connection from the clienthost bus adapter 216 to a server host bus adapter). The inquiryinformation can also include, for example, a vendor or provider of theserver and a SCSI device type. The client OS SCSI service 211 can thenstore the inquiry information (e.g., at a cache and/or at storage 217)such that the stored inquiry information is accessible by othercomponents of the client 200.

In one embodiment, the client OS SCSI service 211 includes other layersso that SCSI requests and responses can be sent and received over a FCnetwork. The client OS SCSI service 211 can include one or more drivers,such as a driver for the client HBA 216, to present devices advertisedover the FC network as standard SCSI devices, which can then bediscovered as such. Thus, the one or more drivers can package SCSIrequests as FC frames and unpackage SCSI responses from FC frames androute the SCSI responses accordingly, such as by implementing FibreChannel Protocol.

All or part of the client OS SCSI service 211 can be included in anoperating system (not shown) that can be operable to initiate theexecution of the instructions provided by components 211-214, interactwith the user at the user interface 220 (e.g., by providing a graphicaluser interface or command line interface and receiving user input),and/or manage hardware (not shown). The operating system may be adaptedto perform other operations across the components of the client 200including threading, resource management, data storage control and othersimilar functionality.

The data optimization module 212 can identify data (e.g., data atstorage 217) that is to be sent to a server and communicate with the FCtransport adapter 214 to send or receive data. In one embodiment, thedata optimization module 212 provides an application programminginterface (API), a dynamic link library (DLL), or other communicativeresource that manages or otherwise handles data send and receiveoperations to be communicated to the server. The data optimizationmodule 212 can be operable to optimize the communication speed betweenthe client 200 and the server, such as by providing data compression anddeduplication operations. The data optimization module 212 can identifynew or modified data at the client 200 (e.g. data at storage 217) thatis to be backed up and/or archived at the server. Additionally, the dataoptimization module 212 can identify data for the client 200 at theserver (e.g., data at a storage unit 180 of server 150).

In some embodiments, data send and receive requests are provided to thedata optimization module 212 from the data storage module 213 at theclient 200. Accordingly, the data optimization module 212 cancommunicate with the data storage module 213 in response to the datasend and receive requests. The data storage module 213 can be, forexample, an application or application suite for backing up and/orarchiving data, such as an enterprise-level backup and recovery suite.The data storage module 213 can be configured to specify that data sendand receive operations are to be sent to a server over a FC network,such as by having a server identifier or other indicator (e.g., a storedvalue, a value received as user input, etc.) indicating that data is tobe transmitted over the FC network. In some embodiments, some or all ofthe functionality provided by the data optimization module 212 iscombined with the data storage module 213.

To facilitate communication between the data optimization module 212 anda server, the data optimization module 212 can provide a call message tothe FC transport adapter 214. The data optimization module 212 canprovide a call message to the FC transport adapter 214 for a variety ofreasons, such as in response to user input that requires serverfunctionality. In one embodiment, a call message is provided to the FCtransport adapter 214 in response to or in anticipation of a data sendand receive requests from the data storage module 213. However, the dataoptimization module 212 can provide a call message to the FC transportadapter 214 for a variety of reasons, and the call message is notlimited to backup and/or storage applications. A call message can be,for example, a message requesting a subroutine or procedure to executeat a server process (e.g., a process 315 of server 300), such as aremote procedure call (RPC) message. Additionally, a call message caninclude data (e.g., data from storage 217, data from a module 212-213,or other data) that is to be sent to the server over the FC network.Thus, the call message can be of any size (e.g., greater than aterabyte). A call message can be, for example, a message to read datafrom or write data to the server, or a message to retrieve informationabout data stored at the server. The data optimization module 212 caninclude data in the call message by, for example, marshaling the data.Correspondingly, the data optimization module 212 can get data in areply message by unmarshaling. In some embodiments, the dataoptimization module 212 concatenates a plurality of data send and/orreceive requests, into one call message. For example, one call messagecan include RPC requests to write data and read data.

The data optimization module 212 can provide a call message to the FCtransport adapter 214 to be sent to a server over the FC network. Toidentify a server process for which the message is intended, the dataoptimization module 212 can provide a process descriptor for theintended server process. The data optimization module 212 can have theseprocess descriptors stored or can provide a call message to the FCtransport adapter 214 for a server that is to get a process descriptorfor a server process. A call message to get a process descriptor can be,for example, a call message intended for a port mapper process at theserver. The FC transport adapter 214 can also be provided a serveridentifier for the server. The data optimization module 212 can providethe server identifier, such as from a stored value or from the datastorage module 213.

The FC transport adapter 214 is operable to receive a call messageprovided by the data optimization module 212 and adapt the call messagefor communication to a server over a FC network. The FC transportadapter 214 can adapt the call message to a SCSI request: a call SCSIrequest. In one embodiment, the FC transport adapter 214 creates a SCSIwrite request that can include the call message as the request'spayload. The FC transport adapter 214 can then identify a connectionfrom the client 200 to the server that is suitable to send the call SCSIrequest across. To retrieve a reply message to a call message (e.g., aRPC reply message) from the server, the FC transport adapter 214 cancreate another SCSI request: a reply SCSI request. A reply SCSI requestcan be a SCSI read request, which the FC transport adapter 214 thensends to the server over the FC network. The FC transport adapter 214can create the reply SCSI request in response to a request from the dataoptimization module 212.

In some embodiments, a call SCSI request includes a plurality ofmessages. The FC transport adapter 214 can receive a plurality of callmessages from the data optimization module 212, such as from a stagingarea and/or storage 117. Accordingly, a call SCSI request can be createdthat includes the plurality of call messages. Similarly, a reply SCSIresponse can include a plurality of reply messages. In one embodiment, acall message can be segmented before communication. Consequently,adapting a call message to be sent to the server over the FC network canrequire a plurality of call SCSI requests, such that each call SCSIrequest contains a segment of the call message. Likewise, a replymessage can be received as a segment of a whole and the FC transportadapter 214 can provide the reply message to the data optimizationmodule 212 as it becomes available.

Accordingly, it is to be understood that call messages and replymessages are not necessarily discrete data requests and responses (e.g.,RPC requests and responses) having a one-to-one relationship with SCSIrequests and responses and sequentially exchanged. For example, the FCtransport adapter 214 can create a plurality call SCSI requests having aplurality of call messages segmentally distributed across the call SCSIrequests and send the plurality of call SCSI requests to the server. TheFC transport adapter 214 can create one or more reply SCSI requests and,in response, receive one or more reply SCSI responses having a pluralityof reply messages segmentally distributed across the one or more replySCSI responses. Therefore, references to a call message or a replymessage adapted to a respective SCSI request or SCSI response (e.g.,included in a payload of a SCSI request or response) can denote themessage data contained in that particular SCSI request or response, andnot necessarily a single or complete RPC request or response.

The FC transport adapter 214 can identify a connection to a server overa FC network using the one or more LUNs discovered by the client OS SCSIservice 211. In one embodiment, the FC transport adapter 214 is operableto examine inquiry information for one or more discovered LUNs of one ormore SCSI devices advertised by the server. Where the client OS SCSIservice 211 has not stored inquiry information for a SCSI device entry,the FC transport adapter 214 can be operable to send a SCSI inquiryrequest for the LUN and receive the inquiry information for that LUN asa response. The FC transport adapter 214 can determine, using theinquiry information, which LUN(s) advertised by the server can receiveSCSI requests over the FC network. In one embodiment, the FC transportadapter 214 makes this determination by examining one or more specificfields of the inquiry information, such as the vendor identification,device identification and/or device type, and verifying that thosespecific fields match predetermined values for those fields. The FCtransport adapter 214 can establish a virtual connection with the serverusing a LUN indicating that it can receive SCSI requests from the client200 via the FC network.

The FC transport adapter 214 can establish a connection for one or moremessages as a virtual connection. In some embodiments, a virtualconnection abstracts the connection from the client 200 to a server to aconnection for one or more messages that are to be communicated betweenthe data optimization module 212 and a server process at the server. Insome embodiments, the FC transport adapter 214 is operable to receive aprocess descriptor provided by the data optimization module 212 and,accordingly, establish the virtual connection using the processdescriptor. The FC transport adapter 214 can associate a virtualconnection with the data optimization module 212. In some embodiments, avirtual connection is associated with the data optimization module 212by, for example, mapping the virtual connection to the data optimizationmodule 212.

A virtual connection can be identified by a virtual connectionidentifier, such as a value. The virtual connection identifier can bepart of a tuple to guarantee the virtual connection is identifiablyunique across space and time; for example, the tuple can include ageneration number and/or a verifier value generated by the server sothat virtual connection identifier can be recycled. The virtualconnection identifier is included in most SCSI requests and SCSIresponses for the virtual connection. For example, a SCSI request caninclude the virtual connection identifier in the logical block address(LBA) field of the SCSI request's command descriptor block (CDB). Forsome SCSI requests, additional parameters (e.g., a virtual connectiontuple and/or a sequence number) can be included in a header added toSCSI request's payload. For call SCSI requests, the call message can beincluded in the SCSI request's payload. Other SCSI requests, such as aSCSI read request to retrieve a reply message, may only include aportion of the virtual connection tuple (e.g., the low-order four bitsof a tuple value) for the virtual connection in the LBA field of theSCSI request's CDB. A reply SCSI response received from the server caninclude the virtual connection identifier and the reply message in theresponse's payload. In some embodiments, a reply SCSI request isvalidated at the server and the reply SCSI response is validated at theclient 200.

The FC transport adapter 214 is also operable to track the sent call andreply SCSI requests using counters or other values. In one embodiment, acall sequence number is incremented for each call SCSI request, and areply sequence number is incremented for each reply SCSI request. Eachsequence number is incremented where a SCSI response is received for theSCSI request that does not indicate the SCSI request failed (e.g.,aborted at the server or failed during the communication of the SCSIrequest over the FC network). Thus, the call sequence number isincremented even where the server only accepts a portion, or none, ofthe call message (e.g., due to insufficient memory at the server).Similarly, the reply sequence number is incremented even where a replySCSI response includes an incomplete reply message or indicates that noreply message is available at the server. The call and reply sequencenumbers can be included in the respective call and reply SCSI requests.However, some SCSI requests (e.g., reply SCSI requests) may only includea bit segment of the sequence number. To acknowledge to the server thata reply SCSI response has been received by the FC transport adapter 214,the FC transport adapter 214 can include in a call SCSI request thereply sequence number of the last reply SCSI request for which a replySCSI response was received.

In one embodiment, some SCSI responses received from the server includethe sequence number. For example, reply SCSI responses include thesequence number in a payload of the reply SCSI response. However, theserver does not increment the sequence numbers included in the SCSIresponses. For reply SCSI responses, the FC transport adapter 214 canvalidate a reply SCSI response by comparing the sequence number includedin the reply SCSI response to the actual sequence number for the replySCSI request. In instances in which the sequence numbers do not match,the FC transport adapter 214 closes the virtual connection and/ordiscards the reply message.

The FC transport adapter 214 can retry failed or aborted SCSI requestswithout incrementing the sequence number. For example, the FC transportadapter 214 can retry a SCSI request where the FC transport adapter 214receives an indication that the SCSI request failed or where a timeoutfor the SCSI response expires. The FC transport adapter 214 can use thesame sequence number for a subsequent SCSI request that recreates thefailed SCSI request to ensure that the client's sequence number matchesthe server's expected sequence number. The FC transport adapter 214 thentransmits the recreated SCSI request to the server over the FC network.

The FC transport adapter 214 can additionally identify the transportpath for a call SCSI request or a reply SCSI request. The transport pathis a path over the FC network between the client 200 and the server,such as a connection between the client host bus adapter 216 and theserver host bus adapter 330. In some embodiments, the transport pathincludes a physical component and a logical component. The physicalcomponent includes the physical path between the client HBA 216 and aserver HBA, such as the server HBA 330. The physical path can include,for example, respective World Wide Names for the client HBA 216 and theserver HBA 330, such as a World Wide Port Name (WWPN) for a port of theclient HBA 216 of the client 200 and the WWPN for a port of the HBA 330of the server 300. In one embodiment, World Wide Node Names (WWNN) canbe included. The logical component can include a LUN or other identifierfor a SCSI device advertised by the server.

In one embodiment, the transport path is identified by issuing a SCSIrequest for one of the SCSI device entries and the SCSI response caninclude the transport path in its payload. The FC transport adapter 214can use any suitable SCSI device entry as the transport path. In someembodiments, the FC transport adapter 214 identifies the transport pathin response to a SCSI response from the server. The FC transport adapter214 can create a SCSI request to be sent over a FC network and providethe transport path to the client OS SCSI service 211, which will thenuse that transport path to the server. For example, all SCSI requestsissued for one SCSI device entry are sent by the client OS SCSI service211 to the same LUN advertised at the same port of one server host busadapter.

The client host bus adapter 216 is operable to perform the physicaltransmission of the SCSI requests and SCSI responses between the client200 and a server HBA of a server (e.g., the server host bus adapter 330of the server 300). One or both of the HBAs 216, 330 can be FibreChannel interface cards. Each HBA 216, 330 has a World Wide Name (WWN)for the respective HBA—a node WWN (WWNN), which is shared by all portson a respective HBA 216 or 330—and a port WWN (WWPN), which is unique toeach port of a respective HBA 216 or 330. As described above, the FCtransport adapter 214 can provide the transport path to the client OSSCSI service 211. Accordingly, the client OS SCSI service 211 uses theclient HBA 216 to send a SCSI request over the FC network using theprovided transport path (or the physical component therein). Note thatalthough only one client HBA 216 is illustrated, the client 200 can havemore than one client HBA. Furthermore, the client HBA 216 can have morethan one port (either physical or virtual). Multiple client HBAs and/ormultiple ports of the same client HBA can be connected with multipleports (either physical or virtual) of one or more server HBAs (e.g.,server HBA 330) at the server.

Turning now to FIG. 3, a simple embodiment of a server 300 is shown. Theserver 300 can be or can include the server 150 of FIG. 1 and can becoupled with one or more local or remote storage units (e.g., thestorage units 180 a-180 b). The server 300 includes, but is not limitedto, several components: including main memory 310, a processor 335, anda server host bus adapter 330. These components may be communicativelycoupled through a bus 340. The bus 340 can be any subsystem adapted totransfer data within the server 300. The bus 340 can be a plurality ofcomputer buses and include additional circuitry to transfer data.

The server host bus adapter 330 is operable to receive the physicaltransmission of SCSI requests over the FC network 130 from a client.Though only one server HBA 330 is illustrated, a server 300 can havemore than one server HBA. Furthermore, the server HBA 330 can have morethan one port (either physical or virtual). Multiple server HBAs and/ormultiple ports of the same server HBA can be connected with multipleports (either physical or virtual) of one or more client HBAs at aclient.

The processor 335 can be any processor suitable to execute instructionsof the components 315-325 stored in main memory 310. Accordingly, theprocessor 335 can be, for example, a central processing unit (CPU), amicroprocessor, or other similar processor. In some embodiments, theprocessor 335 includes a plurality of processors, such as a dedicatedprocessor (e.g., a graphics processing unit), a network processor, orany processor suitable to execute operations of the server 300 connectedwith a client by a Fibre Channel network.

Main memory 310 may be coupled to the processor 335. In someembodiments, main memory 310 provides storage of computer readableinstructions, data structures, program modules, and other data for theserver 300. Main memory 310 can include, but is not limited to, one ormore processes 315 a-315 b, a server Fibre Channel (FC) adapter 320, anda server OS SCSI service 325.

The server OS SCSI service 325 can include, but is not limited to,components operable to handle SCSI requests and responses using theserver HBA 330, such as the SCSI layers (e.g., SCSI interconnect layer,SCSI transport layer, other SCSI layers) and interrelated elements toappropriately route received SCSI requests and send SCSI responses for aclient over a FC network.

In one embodiment, the server OS SCSI service 325 is operable to managethe fundamental SCSI-over-Fibre Channel configuration at the server 300.The server OS SCSI service 325 can provide hardware management of theserver HBA 330 and the transport path between a client and the serverHBA 330, and can therefore include one or more drivers (e.g., atarget-mode driver to provide a data path between the client and otherSCSI layers of the server OS SCSI service 325, and/or a virtual host busdriver to route SCSI requests from the server OS SCSI service 325 to theserver FC adapter 320). One such driver can be for the server HBA 330,so that SCSI devices can be advertised over a FC network. This drivercan package SCSI responses as FC frames and unpackage SCSI requests fromFC frames and route the SCSI requests accordingly, such as byimplementing Fibre Channel Protocol. Additionally, the server OS SCSIservice 325 can provide logical management, such as mapping advertisedLUNs, managing the namespace of one or more LUNs, and routing SCSIrequests.

In one embodiment, the server OS SCSI service 325 is operable to receiveSCSI requests and provide those SCSI requests to the server FC adapter320. Additionally, the server OS SCSI service 325 is operable to receiveSCSI responses from the server FC adapter 320 and send the SCSIresponses to a client over a FC network in response to a SCSI requestfrom the client. The server OS SCSI service 325 can also implement someSCSI functionality, such as sending a SCSI response to a SCSI reportLUNs request.

In one embodiment, the server OS SCSI service 325 advertises one or moreLUNs to a client over a FC network. A LUN can be advertised at one ormore ports of the server HBA 330 and/or at other HBAs (not shown). A LUNcan be mapped to a SCSI device created by the server FC adapter 320.Accordingly, SCSI requests to such a LUN can be routed to the server FCadapter 320. The server FC adapter 320 can specify the advertisement ofLUNs by the server OS SCSI service 325, such as by specifying a clientto which the LUN is to be advertised or specifying a port of the serverHBA 330.

All or part of the server OS SCSI service 325 can be included in anoperating system (not shown) that is operable to initiate the executionof the instructions provided by components 315-320 and/or managehardware (not shown). The operating system may be adapted to performother operations across the components of the server 300 includingthreading, resource management, data storage control and other similarfunctionality.

With respect to the processes 315 a-315 b, a process 315 can be, forexample, an instance of a program at the server 300, such as a set ofmachine-readable instructions that are executed by the processor 335. Aprocess can be a file system process (e.g., a read/write process fordata stored at a storage unit 180). Multiple processes can runconcurrently at the server 300. For example, a file system may havedifferent processes 315 a-315 b. Additionally, multiple processes 315a-315 b can accommodate multiple clients that are concurrently connectedwith the server 300.

Preferably, each process 315 a-315 b has a descriptor associated with itat the server 300. In one embodiment, the descriptor is a port number,and a port for a process 315 can be maintained by a port map.Additionally, a process (e.g., the process 315 a) can identify otherprocesses (e.g., the other process 315 b), such as by providing a portmap. A process 315 can service call messages from the server FC adapter320 that originated at a client, such as by unmarshaling the callmessage, marshaling data in response to the call message (e.g., a replymessage), and/or writing data from the call message to a storage unit(e.g., a storage unit 180). The process 315 can then send a replymessage to the server FC adapter 320. To receive call messages and sendreply messages, the server 300 can provide a server messaging service sothat the messages are communicated between the server and the clientover the FC network.

The server FC adapter 320 can receive SCSI requests from and provideSCSI responses to the server OS SCSI service 325. Thus, the server FCadapter 320 handles, among other SCSI requests, the SCSI write, SCSIread and SCSI inquiry requests from a client. To that end, the server FCadapter 320 implements the server side of a virtual connection with aclient. In some embodiments, the server FC adapter 320 creates one ormore SCSI devices, which then are mapped to one or more LUNs. A devicecan be of any type, such as a processor SCSI device, or any other SCSIdevice type, such as a communications SCSI device. The LUNs are thenadvertised to a client, as described above.

Importantly, because the server FC adapter 320 creates a SCSI device sothat SCSI requests for the associated LUN are routed to the server FCadapter 320, instead of to a logical disk or physical device, the LUN iseffectively a rendezvous point at the server FC adapter 320 for SCSIrequests sent to the server 300 over a FC network. Consequently, theserver FC adapter 320 can accept multiple client SCSI requests to asingle LUN. Furthermore, this allows the LBA field of SCSI requests toinclude values that are not an actual logical block address. Forexample, the LBA field can include the virtual connection identifierinstead of an actual logical block address for the created SCSI device.

The server FC adapter 320 can implement the SCSI inquiry request sent bya client to describe an advertised LUN by, for example, responding witha SCSI response indicating that the LUN can receive SCSI requests over aFC network. Thereafter, the server FC adapter 320 can receive one ormore SCSI requests over the FC network from the client to establish avirtual connection. The server FC adapter 320 can respond to suchrequests by assigning a virtual connection identifier for the virtualconnection. The virtual connection identifier can be part of a tuple toguarantee the virtual connection is identifiably unique across space andtime; for example, the tuple can include a generation number and/or averifier value generated by the server FC adapter 320. Additionally, theserver can identify a transport path over the FC network that the clientis to use for the virtual connection by, for example, selecting thetransport path from a catalog of transport paths provided by the client.

The server FC adapter 320 can also associate a process 315 with thevirtual connection by, for example, using a process descriptor for theprocess 315 provided in a SCSI request for a virtual connection from aclient. Once a virtual connection is established, the server FC adapter320 is able to handle SCSI requests that include the virtual connectionidentifier using the associated process 315. The server FC adapter 320can provide call messages to a process 315 by, for example, extracting acall message from a call SCSI request originating at a client andproviding the call message to the process 315. Thereafter, the server FCadapter 320 can respond to the call SCSI request with a status code(e.g., a SCSI status code or a vendor-specific status code) indicatingthe all or part of the call message has been accepted.

In response to the call message, a process 315 can provide a replymessage to the server FC adapter 320. Where the server FC adapter 320subsequently receives a reply SCSI request for the virtual connection,the server FC adapter 320 can respond by creating a reply SCSI responsethat includes the virtual connection identifier and the reply message inthe payload.

In one embodiment, the server FC adapter 320 can associate a process 315with a virtual connection by establishing a backend connection from theserver FC adapter 320 to a process 315. This connection can be, forexample, a localhost connection or other transmission control protocol(TCP) connection established using a process descriptor (e.g., a portnumber) for a process 315. Accordingly, server FC adapter 320 canassociate the virtual connection identifier with the process 315 usingthe backend connection.

The server FC adapter 320 is also operable to monitor the expectedreceived call and reply SCSI requests using counters or other values. Anexpected sequence number can be included in a SCSI response from theserver FC adapter 320. In one embodiment, an expected call sequencenumber is incremented for each call SCSI response to a received callSCSI request, and an expected reply sequence number is incremented foreach reply SCSI response to a received reply SCSI request. Each expectedsequence number is incremented after a SCSI response is provided to theserver OS SCSI service 325 to be sent to a client over a FC network. Arespective expected call sequence number is incremented even where theserver FC adapter 320 only accepts a portion, or none, of a call message(e.g., due to insufficient memory at the server). Similarly, the replysequence number is incremented even where the server FC adapter 320 onlyincludes an incomplete reply message, or returns an indication that noreply message is available.

SCSI requests received at the server FC adapter 320 from a client caninclude the sequence number, or a portion thereof. For example, callSCSI request includes the call sequence number in a payload of the replySCSI response. However, the server FC adapter 320 does not increment thesequence numbers included in the SCSI responses; rather, the expectedsequence numbers are only incremented after the SCSI responses areprovided to the server OS SCSI service to be sent to a client over a FCnetwork.

The server FC transport adapter 320 can validate SCSI requests receivedfrom a client according to the actual sequence numbers included in theSCSI requests. For SCSI requests that include the full sequence number(e.g., call SCSI requests), the server FC transport adapter 320 canvalidate the SCSI request by comparing the sequence number included inthe call SCSI request to the expected call sequence number. For SCSIrequests that include only a portion of the sequence number (e.g., replySCSI requests), the server FC transport adapter 320 can validate a replySCSI request by comparing the portion of the sequence number included inthe reply SCSI request to the corresponding portion of the expectedreply sequence number. The server FC adapter 320 validates a sequencenumber included in a SCSI request that matches the excepted sequencenumber. In some embodiments, the server FC adapter 320 validates asequence number included in a SCSI request that indicates a retried SCSIrequest (e.g., the expected sequence number is an increment greater thanthe actual sequence number in the SCSI request). In instances in whichthe sequence numbers do not match and do not indicate a retried SCSIrequest, the SCSI request is erroneous and may be discarded or respondedto with an indication that the sequence number is erroneous.

Because a client can retry failed or aborted SCSI requests withoutincrementing the sequence number, the server FC adapter 320 is operableto handle situations in which the sequence number included in the SCSIrequest indicates a retried SCSI request. For retried call SCSImessages, the server FC adapter 320 again responds with a call SCSIresponse indicating that all or part of the call message from the callSCSI request has been accepted. For retried reply SCSI messages, theserver FC adapter 320 again responds with a reply SCSI responseincluding all or part of the reply message, which may be stored in abuffer or cache until the server FC adapter 320 receives anacknowledgement from the client that the reply SCSI message has beenreceived by the client. The server FC adapter 320 then sends the storedSCSI response to the client over a FC network.

It should be appreciated that embodiments of the invention as will behereinafter described may be implemented in conjunction with theexecution of instructions by a processor (e.g., processor 218 orprocessor 335) of a client 110 or the server 150 and/or other circuitryof a client 110 or the server 150. Particularly, circuitry of both aclient 110 and the server 150, including but not limited to a respectiveprocessor can operate under the control of a program, routine, or theexecution of instructions to execute methods or processes in accordancewith embodiments of the invention. For example, a data optimizationmodule at a client 110 may be implemented in firmware, software (e.g.,stored in main memory) or hardware and may be implemented by a processorand/or other circuitry of the client 110. Further, it should beappreciated that the terms processor, microprocessor, circuitry,controller, etc., refer to any type of logic or circuitry capable ofexecuting logic, commands, instructions, software, firmware,functionality and the like.

FIG. 4A illustrates a conceptual block diagram of message communicationbetween a client 410 and a server 450 using SCSI requests over a FCnetwork 430. The client 410 can be or can include the client 200 of FIG.2 and, accordingly, the data storage module 413 can be the data storagemodule 213, the data optimization module 412 can be the dataoptimization module 212, the FC transport adapter 414 can be the FCtransport adapter 214 and the client OS SCSI server 411 can be theclient OS SCSI server 211. The server 450 can be or can include theserver 300 of FIG. 3 and, accordingly, the process 452 can be a process315 a-315 b, the server FC adapter 454 can be the server FC adapter 320and the server OS SCSI service 455 can be the server OS SCSI service325. Illustrative embodiments of methods for the system 400 of FIG. 4Aare described at FIGS. 5-14.

Beginning first with the server 450, the server FC adapter 454 createsone or more SCSI devices, which are mapped to one or more LUNs to beadvertised to the client 410. As described above, the LUN is effectivelya rendezvous point at the server FC adapter 454 for SCSI requests sentto the server 450 over the FC network 430. Consequently, the server FCadapter 454 can accept multiple client SCSI requests from multipleclients to a single LUN. Furthermore, this allows the LBA field of SCSIrequests to include values that are not an actual logical block address.For example, the LBA field can include the virtual connection identifierand low-order bits of the actual sequence number. The server OS SCSIservice 455 advertises the created LUN to the client 410 over the FCnetwork 430.

Turning to the client 410, the client SCSI OS service 411 discovers anadvertised LUN as a SCSI device and identifies the LUN as such, e.g., bycreating a SCSI device entry. The FC transport adapter 414 can thenexamine the discovered LUN to determine if the LUN is a transport pathover the FC network 430 to the server FC adapter 454, such as by sendinga SCSI read request to retrieve server information.

To communicate a message using SCSI requests and responses, the dataoptimization module 412 can provide a process descriptor to the FCtransport adapter 414. The process descriptor identifies a serverprocess 452 to which the module 412 is attempting to communicate a callmessage. The FC transport adapter 414 can then establish a virtualconnection by receiving a virtual connection identifier for the virtualconnection and by sending a SCSI request to the discovered LUN thatincludes the process descriptor. The server OS SCSI service 455 receivesthe SCSI request for the LUN and routes the SCSI request to the serverFC adapter 454. Using the process descriptor in the SCSI request, theserver FC adapter associates the virtual connection with the process452.

After providing the process descriptor, the module 412 provides the callmessage to the FC transport adapter 414. In one embodiment, the callmessage is provided in response to a data send or receive request fromthe data storage module 413. The FC transport adapter 414 adapts thecall message to be communicated over the FC network 430 as a SCSIrequest by, for example, creating a call SCSI request that includes thevirtual connection identifier in the LBA field of the SCSI request andthe call message in a payload of the SCSI request. In some embodiments,the payload includes a header added by the FC transport adapter 414 thatincludes other parameters (e.g., a virtual connection tuple, full callsequence number, etc.). The FC transport adapter 414 then sends the callSCSI request over the FC network 430 to the discovered LUN using theclient OS SCSI service 411.

The call SCSI request is then received over the FC network 430 by theserver OS SCSI service 455. The server OS SCSI service 455 routes thecall SCSI request to the server FC adapter 454. The server FC adapter454 receives the call SCSI request and examines LBA field of the callSCSI request's CDB to validate or identify the virtual connection. Oncethe server FC adapter 454 has validated the virtual connection, theserver FC adapter extracts the call message from the call SCSI request,such as by separating it from the SCSI-specific data (e.g., the CDB) andfrom the header included in the request payload. The server FC adapter454 then provides the call message to the server process 452. Inresponse to the call message, the process 452 services the call messageand provides a reply message to the server FC adapter 454.

Thus, the call message traverses the call message path 403 as a callmessage that is adapted to a SCSI request, sent over the FC network 430,extracted from the SCSI request, and then provided to the intendedprocess 452. To retrieve a reply message to the call message, the FCtransport adapter 414 creates a reply SCSI request. The reply SCSIrequest includes the virtual connection identifier in the LBA field ofthe SCSI request. The FC transport adapter 414 then sends the reply SCSIrequest over the FC network 430 to the discovered LUN using the clientOS SCSI service 411. In some embodiments, the reply SCSI request iscreated in response to a request from the data optimization module toget the reply message. Additionally, the FC transport adapter 414 cancreate and send a plurality of SCSI requests having one or more callmessages before creating and sending the reply SCSI request.

The server OS SCSI service 455 then receives the reply SCSI request overthe FC network 430. The server OS SCSI service 455 routes the reply SCSIrequest to the server FC adapter 454. The server FC adapter 454 receivesthe reply SCSI request and examines the LBA field of the reply SCSIrequest's CDB to validate or identify the virtual connection. Once theserver FC adapter 454 has validated the virtual connection, the serverFC adapter 454 adapts the reply message to a SCSI response. The serverFC adapter 454 adapts the reply message to be communicated over the FCnetwork 430 as a SCSI response by, for example, creating a reply SCSIresponse that includes the reply message in a payload of the SCSIresponse. The response payload can include a header added by the serverFC adapter 454 that includes the virtual connection identifier and/orother parameters (e.g., a virtual connection tuple, reply sequencenumber, etc.). The server FC adapter 454 then responds to the reply SCSIrequest by sending the reply SCSI response over the FC network 430 usingthe server OS SCSI service 455.

The client OS SCSI service 411 then receives the reply SCSI responseover the FC network 430. The client OS SCSI service 411 routes the replySCSI response to the FC transport adapter 414. The FC transport adapter414 receives the reply SCSI response and examines the header of theresponse's payload to validate or identify the virtual connection. Oncethe FC transport adapter 414 has validated the virtual connection, theFC transport adapter 414 extracts the reply message from the reply SCSIresponse, such as by separating it from the SCSI-specific data and fromthe header included in the response payload. The FC transport adapter414 then provides the reply message to the module 412. Thus, the replymessage traverses the reply message path 404 as a reply message that isadapted to a SCSI response, sent over the FC network 430, extracted fromthe SCSI response, and then provided to the module 412.

To illustrate the communication between the client 410 and the server450 of FIG. 4A, FIGS. 4B and 4C show embodiments of structures of SCSIrequests and responses communicated over the FC network 430. FIG. 4Bshows a SCSI request 4110 that can be a SCSI write request or a SCSIread request. In the latter case, the SCSI request 4110 does not includea payload 4115. The SCSI request 4110 is packaged as a Fibre Channelframe 4100 (or multiple frames, if appropriate) that includes atransport path 4101 between the client 410 and the server 450 along theFC network 430. In one embodiment, the FC transport adapter 414specifies the transport path 4101 by issuing the SCSI request 4110 tothe client-side SCSI device entry for the discovered LUN. Consequently,the FC frame 4100 having the SCSI request 4110 traverses the FC network430 from the client 410 to the server 450 according to the physicalcomponent 4103. Once received by the server 450, the server OS SCSIservice 455 can route the SCSI request 4110 to the server FC adapter 454according to the logical component 4102.

Prior to being packaged as the FC frame 4100, the FC transport adapter414 can create the SCSI request 4110. For many SCSI requests, the FCtransport adapter 414 adopts the LBA field 4112 of the SCSI protocol tocontain the virtual connection identifier for the virtual connectionbetween the client 410 and the server 450 over the FC network 430.However, the LBA field 4112 of an initial SCSI request to begin theestablishment of a virtual connection can instead include an indicationthat the client 410 wishes to establish a virtual connection with theserver 450 (e.g., using a predetermined value or flag). In otherembodiments, the FC transport adapter 414 can likewise omit a virtualconnection identifier from the LBA field 4112 of a SCSI request for anoperation not requiring a virtual connection (e.g., a SCSI request tolog a message at the server FC adapter 454). Additionally, the FCtransport adapter 414 can include other parameters in the LBA field 4112beyond a virtual connection identifier or an indicator.

In some embodiments, such as SCSI read requests, the SCSI request 4110may not include any parameters outside of those added to a CDB 4111 bythe FC transport adapter 414. In other embodiments of the SCSI request4110, such as SCSI write requests, the FC transport adapter 414 createsthe payload 4115. The FC transport adapter 414 can include a header 4116in the payload 4115. The content of the header 4116 varies according tothe embodiment of the SCSI request 4110. For example, the FC transportadapter 414 can add parameters to the header 4116 such as a processdescriptor, a catalog of transport paths, a virtual connection tuple, arequest type (e.g., a code for virtual connection establishment or tosend a call message), or, if applicable, information about a callmessage 4117 (e.g., a byte size, a byte sequence number, or a callsequence number) or an acknowledgement that a previous SCSI responsesent by the server 450 has been received (e.g., a reply sequence numberof a last-received reply SCSI response). Additionally, the FC transportadapter 414 can include all or part of a call message 4117, such as acall message provided by the data optimization module 412. The createdSCSI request 4110 can then be packaged as the FC frame 4100 and sent tothe server 450 over the FC network 430.

At the server 450, the server FC adapter 454 examines the LBA field 4112included in the CDB 4111 of the SCSI request 4110. The server FC adapter454 can use the included virtual connection identifier either alone orin combination with other parameters of the SCSI request 4110, such as aSCSI operation code 4113, to validate or handle the SCSI request 4110.Where included, the server FC adapter 454 can also examine the header4116 for validation and handling. For call SCSI requests, the server FCadapter 454 can extract the call message 4117 and provide the callmessage 4117 to the server process 452.

In response to a client SCSI request, the server 450 can send a SCSIresponse 4210 to the client 410 over the FC network 430, as shown inFIG. 4C. Similar to the SCSI request 4110, the SCSI response 4210 ispackaged as a FC frame 4200 (or multiple frames, if appropriate) to besent to the client 410 along the FC network 430. In one embodiment, thetransport path 4201 is the same as that of the client SCSI request towhich the SCSI response 4210 is responsive. Once received by the client410 over the FC network 430, the client OS SCSI service 411 can routethe SCSI response 4210 to the FC transport adapter 414.

For many SCSI responses, the server FC adapter 454 creates the SCSIresponse 4210 prior to its being packaged as the FC frame 4200. In someembodiments, the server FC adapter 454 adopts the sense data 4113 of theSCSI protocol to contain a status code 4214 responsive to the clientSCSI request. For example, the status code 4214 can indicate that theclient SCSI request has been completely or incompletely processed by theserver FC adapter 454, that the client SCSI request has been rejected bythe server FC adapter 454, or other status related to the client SCSIrequest. The status code 4214 can comprise a number of values so that ameaningful status is conveyed; for example, the status code 4214 can bea combination of a generic status code (e.g., “check condition”) and avendor-specific status code (e.g., an indication that only a segment ofa call message has been accepted or an indication that a reply messageis not available). Additionally, the server FC adapter 454 can includefurther information about the status code 4214, such as by including inthe sense data 4213 a vendor-specific value or a number of bytes of acall message that have been accepted.

In some instances, the server FC adapter 454 does not create the SCSIresponse 4210. For example, the client SCSI request may be abortedbefore reaching the server FC adapter 454 and, therefore, the SCSIresponse 4210 includes an “aborted” status code 4214 originating at acomponent that aborted the client SCSI request (e.g., the server OS SCSIservice 455).

In some embodiments, such as SCSI responses to call SCSI requests fromthe client 410, the SCSI response 4210 may not include any parametersoutside of the status code 4214 or the sense data 4213. In otherembodiments of the SCSI response 4210, the server FC adapter 454 createsa payload 4215. The server FC adapter 454 can include a header 4211 inthe payload 4215. The header 4211 can include a virtual connectionidentifier 4212 as well as other parameters, such as a virtualconnection tuple, a request type (e.g., a code for virtual connectionestablishment or to send a reply message), a request for the client 410to migrate to a new transport path or, if applicable, information abouta reply message 4217 (e.g., a byte size, a byte sequence number, or areply sequence number). Additionally, the server FC adapter 454 caninclude all or part of a reply message 4217, such as a reply messageprovided by the server process 452. The created SCSI response 4210 canthen be packaged as the FC frame 4200 and sent to the client 410 overthe FC network 430.

At the client 410, the FC transport adapter 414 examines the status code4214 included in the sense data 4213 of the SCSI response 4210. The FCtransport adapter 414 can use the status code 4214 either alone or incombination with other parameters of the SCSI response 4210 to validateor handle the SCSI response 4210. Where included, the FC transportadapter 414 can examine the header 4211 for validation and handling. Forreply SCSI responses, the FC transport adapter 414 can extract the replymessage 4217 and provide the reply message 4217 to the data optimizationmodule 412.

FIG. 4D illustrates an embodiment of a logical block address field 4300included in a CDB of a client SCSI request that has been created by theFC transport adapter 414 for a virtual connection. In the illustratedembodiment, the LBA field 4300 is divided into five discrete fields: avirtual connection identifier 4301, a sequence number 4302, a generationnumber 4303, a byte padding 4304, and a timeout 4305. Thus, the LBAfield 4300 does not address an actual location of data or data blocks,but can be handled by the server FC adapter 454 to communicate messagesover the FC network 430 using the SCSI protocol.

In FIG. 4D, the virtual connection identifier 4301 is included toidentify a virtual connection which the FC transport adapter 414 is touse to communicate messages over the FC network 430. The sequence number4302 is included to monitor or validate an associated one of a replymessage or a call message. Similarly, a generation number 4303 of avirtual connection tuple is included to validate the SCSI request. Thebyte padding 4304 is included to indicate a number of padding bytesincluded in SCSI requests and responses. The byte padding 4304 can beincluded in some embodiments because the FC transport adapter 414 andthe server FC adapter 454 use SCSI read and write requests to transferdata of any size, but the SCSI requests operate in units of block thatare customarily 512 bytes. Finally, a timeout 4305 is included toindicate an expected maximum duration for a SCSI request to be servicedor for a SCSI response to be received. In some embodiments, one or moreparameters 4301-4305 can be a bit-segment of a full parameter, such as alow-order bit segment of the sequence number 4302 or the generationnumber 4303.

Importantly, the parameters included in the LBA field 4300 of FIG. 4Dare to be regarded as illustrative and not limiting. Other parametersare used in other embodiments of the LBA field 4300. For example, theLBA field of an initial SCSI request to begin the establishment of avirtual connection can instead include an indication that the client 410wishes to establish a virtual connection with the server 450 (e.g., apredetermined value or flag).

Turning to FIG. 5, a method 500 for initializing a client is illustratedaccording to one embodiment of the invention. The client can beinitialized for connecting with a server over a Fibre Channel network sothat the client can send SCSI requests to and receive SCSI responsesfrom the server. The method 500 can be performed by a client 110 of FIG.1 to connect with the server 150 over the FC network 130.

Beginning first with operation 501, an identifier for the server isreceived at the client. In one embodiment, the server identifier can bereceived as input at the client. For example, a user can input theserver identifier at an interface of the client, such as a graphicaluser interface (GUI) provided by a module at the client or a commandline interface. In some embodiments, the server identifier can be astored value.

At operation 502, a FC transport adapter at the client is initialized.The initialization can occur, for example, when the client boots up, inresponse to a module at the client, or in response to user input. Insome embodiments, operations 502 and 501 are transposed or areconcurrent so that the FC transport adapter is initialized before orsimultaneously with the reception of the identifier for the server.Thus, the client can receive a server identifier using FC transportadapter.

Proceeding to operating 503, the client registers the server. In someembodiments, registering the server with the received server identifierindicates that messages between the client and the server are to becommunicated over a FC network. According to the server identifier, theclient can determine which of the discovered LUNs are paths to theregistered server by, for example, sending a SCSI read request to getserver information and comparing the SCSI response to the serveridentifier. Thus, call messages from a module can be sent to theregistered server using discovered SCSI LUNs.

With the server registered, the client is operable to communicate withthe server using SCSI requests over a FC network. In one embodiment, theclient communicates with the server in response to a module at theclient that is to communicate with a server process. To do so, a virtualconnection is first established. With a virtual connection established,the client is operable to communicate with the server over the FCnetwork using messages adapted to SCSI requests and responses. Theclient can establish additional virtual connections for additionalmessage communication.

Where the client is to end communication with the server, the client canunregister the server at operation 504. This operation 504 can freeresources at the client or allow the client to register another server(n.b., the client can have more than one server registeredconcurrently). Additionally, the client can unregister a server if anerror has been detected (e.g., where the FC network connecting theclient to the server is unavailable or where there is a hardware failureat the client or the server). In one embodiment, the server can beunregistered in response to input (e.g. user input).

At completion, the FC transport adapter at the client is shut down atoperation 505. The shutdown 505 can occur, for example, when the clientshuts down and/or in response to user input. In some embodiments,operations 505 and 504 are transposed or are concurrent so that the FCtransport adapter is shutdown before or simultaneously with theunregistering of the server.

FIG. 6 illustrates an embodiment of a method 600 for establishing avirtual connection by a client connected with a server over a FCnetwork. As described above, the method 600 is performed where theclient wishes to communicate with the server. The method 600 can beperformed in response to a data optimization module, such as where thedata optimization module is to provide a call message. In someembodiments, a FC transport adapter includes instructions to perform themethod 600. For example, the FC transport adapter can be the FCtransport adapter 214 of client 110 illustrated at FIG. 2.

Beginning with operation 601, a process descriptor and a serveridentifier are provided at the client. The process descriptor and theserver identifier may be provided sequentially or simultaneously. Theprocess descriptor and the server identifier can be provided by a moduleof the client, and subsequently be received by a FC transport adapter.The server identifier can identify the registered server with which theclient is to communicate, and the process descriptor can identify aprocess at the registered server for which one or more messages from theclient are intended. The module can provide this information in responseto a data send or receive request from another client module (e.g., adata storage module).

At operation 602, the client catalogs the transport paths to theregistered server using LUNs discovered by the client. The client cancatalog one or more transport paths to the registered server by issuinga SCSI request to get server information for each LUN, such as a SCSIread request. The client can receive a SCSI response for each SCSIrequest that includes a server identifier and the transport path betweenthe client and the server over the FC network. In response to the one ormore SCSI responses, the client can catalog the transport pathscorresponding to the LUNs advertised by the registered server. In oneembodiment, the client compares the server identifier at the client(e.g., the server identifier of the registered server) to a serveridentifier included in the SCSI response. Where the client validates theserver identifier received in the SCSI response, the client catalogsthat transport path to the registered server. The cataloged transportpaths can be stored or cached. Accordingly, the client can catalogtransport paths by using stored or cached transport paths, instead ofissuing SCSI requests.

In one embodiment of operation 602, the client can determine that aserver can receive call SCSI requests over the FC network. The clientcan issue a SCSI inquiry request for each SCSI device entry and receivea SCSI inquiry response. The client can then examine the one or morefields of the SCSI inquiry response that indicate a server can receiveSCSI requests over the FC network. This inquiry information can bestored or cached so that the client may issue further SCSI requests onlyfor LUNs advertised by a server that can receive SCSI requests over theFC network. In one embodiment, this inquiry information is stored orcached by a client OS SCSI service as part of the SCSI device discoveryprocess so that the FC transport adapter can later access the inquiryinformation. The FC transport adapter may also be operable to getinquiry information.

At operation 603, the client creates a first SCSI request for theregistered server that is to start the establishment of a virtualconnection. The SCSI request can be, for example, a SCSI read request.The SCSI request can include an indication that this SCSI request is tostart establishing a virtual connection. For example, the indication canbe a predetermined value included in the LBA field of the SCSI request.

At operation 604, the client sends the first SCSI request to theregistered server over the FC network to start the establishment of avirtual connection. The client can send the SCSI request to theregistered server over the FC network using any transport path to theregistered server, such a selected transport path from the catalog oftransport paths.

At operation 605, the client receives a first SCSI response to the firstSCSI request over the FC network to start the establishment of a virtualconnection with the registered server. The first SCSI response caninclude an identifier for the virtual connection, such as a value.Additionally, the SCSI response can include parameters such as ageneration number and/or a verifier value. The SCSI response can furtherinclude server identification information so that the client can verifythat the SCSI response is from the registered server.

In response to receiving the identifier for the virtual connection, theclient creates a second SCSI request at operation 606. This second SCSIrequest can be, for example, a SCSI write request. The second SCSIrequest can include the virtual connection identifier and the processdescriptor. Additionally, the second SCSI request can include thegeneration number and/or a verifier value from the first received SCSIresponse. In some embodiments, the second SCSI request includes thecataloged transport paths. Some of this information can be included inthe LBA field of the SCSI request, while other information can beincluded in the SCSI request payload.

At operation 607, the client sends the second SCSI request to theregistered server over the FC network to indicate the server processwith which the client is to communicate. The client can send the SCSIrequest to the registered server over the FC network using any transportpath to the registered server, such as the transport path used for thefirst SCSI request.

At operation 608, the client receives a second SCSI response to thesecond SCSI request over the FC network. This second SCSI response canindicate that the registered server is able to establish the virtualconnection. In one embodiment, the SCSI response is a status code, suchas a SCSI status code or vendor-specific status code. The client candetermine whether the registered server is able to establish the virtualconnection based on the second SCSI response.

Where the client determines that the registered server is able toestablish the virtual connection, the client creates a third SCSIrequest at operation 609. This third SCSI request can be a SCSI readrequest. The third SCSI request includes the virtual connectionidentifier. The virtual connection identifier can be included in the LBAfield of the third SCSI request. Additional information can be includedin the third SCSI request, such as in the LBA field.

At operation 610, the client sends the third SCSI request over the FCnetwork to the registered server over the FC network to complete theestablishment of the virtual connection. The client can send the thirdSCSI request to the registered server over the FC network using anytransport path to the registered server, such as the transport path usedfor the first and/or second SCSI request.

At operation 611, the client receives a third SCSI response to the thirdSCSI request over the FC network. The third SCSI response can includethe virtual connection identifier. Additionally, the third SCSI responsecan include a selected transport path that the client is to use for thevirtual connection. In some embodiments, the reception of the third SCSIresponse completes the establishment of the virtual connection.

At operation 612, the client associates the virtual connection with themodule providing the process descriptor. The client can associate thevirtual connection by, for example, mapping the virtual connection tothe module. In some embodiments, operation 612 occurs before some of thepreceding operations of the method 600. For example, the client canassociate the virtual connection at any point after the virtualconnection identifier is received at operation 605. With the virtualconnection associated, the client is operable to send and receivemessages adapted to SCSI requests and responses over the FC network.

In one embodiment of operation 612, the virtual connection is associatedwith the module by establishing a socket connection between the moduleproviding the process descriptor and the FC transport adapter. Forexample, the module can connect a stream socket with the FC transportadapter and provide that process descriptor when connecting. The FCtransport adapter can subsequently map a socket identifier, such as afile descriptor for the socket, with the virtual connection identifier.Accordingly, call messages can be received at the FC transport adapteras writes to the socket from the module. The module can then poll thesocket and use socket reads to receive reply messages or other dataprovided to the socket by the FC transport adapter.

Turning to FIG. 7, a method 700 illustrates an embodiment of a method ata client for communicating messages between the client and a server overa Fibre Channel network using SCSI requests and responses. This method700 can be performed by a FC transport adapter 214 communicativelycoupled with one or both of a data optimization module 212 and a datastorage module 213 of a client 110, as shown in FIG. 2. In someembodiments, the method 700 is performed where a virtual connection hasbeen established with a registered server over the FC network.Accordingly, the SCSI requests include a virtual connection identifierassociated with a client module.

Beginning with operation 701, a call message is received at the client.As described above, this call message can include, for example, an RPCrequest and data corresponding to the RPC request. In some embodiments,the call message is received at the client's FC transport adapter from aclient module, such as a data optimization module, that is associatedwith a virtual connection—i.e., the module providing a processdescriptor used to establish the virtual connection. In one embodiment,the call message is received over a socket connection between the FCtransport adapter and the module. For example, the module writes thecall message to a socket connected to the FC transport adapter.

At operation 702, the client creates a call SCSI request, such as a SCSIwrite request. In one embodiment, the FC transport adapter creates thecall SCSI request to adapt the call message to be sent over the FCnetwork using the SCSI request structure. The virtual connectionidentifier is included in this call SCSI request. For example, the LBAfield of the call SCSI request can include the virtual connectionidentifier. Other parameters can be included in the LBA, such as atimeout value and an operation sequence number (or a bit segment of theoperation sequence number). The call SCSI request includes the callmessage in the request's payload.

In some embodiments, the payload of the call SCSI request includes aheader added by the FC transport adapter in addition to the callmessage. The header can include some parameters for verification andhandling of the call SCSI request and the call message containedtherein. For example, the header can include a virtual connection tuple.The header may additionally include an operation code so that the serverreceiving the call SCSI request can handle the call messageappropriately. Furthermore, the header can include a call sequencenumber, a byte sequence number, a number of bytes requested, or anacknowledgement that a SCSI response has been previously received by theclient, such as a reply sequence number for a preceding reply SCSIrequest.

Subsequently, the client sends the call SCSI request to the server overthe FC network so that the call message may be received by a serverprocess for which it is intended, as shown at operation 703. In oneembodiment, the FC transport adapter provides the call SCSI request to aclient OS SCSI service to be sent over the FC network. The client cansend the SCSI request to the registered server over the FC network usingany transport path to the registered server, such as a transport pathreceived from the server during the establishment of the virtualconnection. In some embodiments, one or more additional call SCSIrequests can be created and sent to the server before proceeding.

At operation 704, the client creates a reply SCSI request, such as aSCSI read request. The reply SCSI request is created to retrieve a replymessage from a server process over the FC network using the SCSI requestand response structure. The virtual connection identifier is included inthis reply SCSI request. For example, the LBA field of the reply SCSIrequest can include the virtual connection identifier. Other parameterscan be included in the LBA field, such as a timeout value, a replysequence number and/or a tuple value of the virtual connection tuple (orbit segments of the reply sequence number or tuple value). In someembodiments, this operation 704 is performed in response to a requestfrom a module, such as the data optimization module that provided thecall message.

Proceeding to operation 705, the client sends the reply SCSI request tothe server over the FC network so that the reply message may beretrieved from the server process for which the call message wasintended. The client can send the SCSI request to the registered serverover the FC network using any transport path to the registered server,such as a transport path received from the server during theestablishment of the virtual connection.

In response to operation 705, the client receives a reply SCSI responsefrom the server over the FC network at operation 706. In someembodiments, the payload of the reply SCSI response includes a header inaddition to the reply message. The header can include some parametersfor verification and handling of the reply SCSI response and the replymessage contained therein. For example, the header can include a virtualconnection identifier or virtual connection tuple so that the clientreceiving the reply SCSI response can validate the response. The headermay additionally include an operation code so that the client receivingthe reply SCSI response can handle the reply message appropriately.Furthermore, the header can include a reply sequence number, a bytesequence number, a number of bytes returned in the reply message, or anumber of additional bytes of the reply message not returned in thereply SCSI response payload but available to be retrieved from theserver. Where the reply SCSI response includes a number of additionalbytes of the reply message not returned in the reply SCSI response, theclient can create one or more additional reply SCSI requests and sendthe one or more reply SCSI requests to the server of the FC network.

In one embodiment of operation 706, the reply SCSI response includes anindication that the server requests that the client migrate to anothertransport path for future SCSI requests. This indication can be, forexample, a flag or Boolean value, or may simply be the presence of a newtransport path. Going forward, the client can use the new transport pathwhen sending SCSI requests to the server.

At operation 707, the reply message is extracted from the reply SCSIresponse. The extraction can include, for example, separating the replymessage from SCSI-specific or FC-specific data. In some embodiments,this operation involves recognizing a header in the reply SCSI responseand separating the header from the reply message. For example, theheader can include a number of bytes of the reply message in the payloadand a padding or offset of the reply message bytes within the payload sothat the FC transport adapter recognizes that number of bytes as thereply message.

With the reply message available at the client, the reply message isprovided to the module associated with the virtual connection atoperation 708. The reply message can be sent to the module or madeavailable so that the module can retrieve the reply message, such as byreading the reply message. In some embodiments, the virtual connectionincludes a stream socket connected between the FC transport adapter andthe module. The FC transport adapter can therefore provide the replymessage to the module by making the reply message available at thestream socket. The module may be polling the socket and, where the replymessage is available, read the socket to retrieve the reply message.Alternatively, the FC transport adapter can write to the socket toprovide the message to the module.

In one embodiment, the client determines that all call messages from theclient module and reply messages from the server process have beensatisfactorily communicated. At this point, the client can close thevirtual connection, such as by creating a SCSI request to close thevirtual connection and sending that SCSI request to the server over theFC network. For example, the client can close the virtual connection inresponse to the closing of the socket (e.g., where the module closes thesocket). The client can then return to the start virtual connectionestablishment state, where a module may provide another processdescriptor indicating that the module is to send additional callmessages to a server process, as shown in FIGS. 6 and 7.

Now with respect to a server implementation of communicating messagesbetween the server and a client over a Fibre Channel network using SCSIrequests and responses, FIG. 8 illustrates a method 800 for initializinga server according to one embodiment of the invention. The server can beinitialized for connecting with a client over a Fibre Channel network sothat the server can receive SCSI requests from and send SCSI responsesto the client. The method 800 can be performed by the server 150 of FIG.1 to connect with a client 110 over the FC network 130.

Beginning first with operation 801, an identifier for the server isreceived at the client. In some embodiments, the server identifier canbe a stored value or received at the server as input (e.g., user input).In one embodiment, the server identifier can be received as input at theclient and sent to the server. For example, a user can input the serveridentifier at an interface of a client, such as a command lineinterface, and communicate the server identifier to the server using acryptographic network protocol, such as Secure Shell or other similarprotocol.

At operation 802, a client group is created. A client group can definethe SCSI devices advertised to a client and at which ports of a serverhost bus adapter those devices are to be advertised. To that end, theserver can add a client to the client group, where the client isconnected to the server over the FC network, as shown at operation 803.

Proceeding to operating 804, the server creates one or more devices forthe client group. The number of devices created may be contingent uponclient considerations, such as whether the client serializes SCSIrequests and responses and presents those requests and responses to asingle client-side SCSI device entry. For other clients, the client candynamically adjust the number of simultaneous SCSI requests andresponses, so that the server need advertise only one SCSI device. EachSCSI device is then mapped to a respective LUN for the client group atoperation 805. In some embodiments, a SCSI device can be added to morethan one client group, so the SCSI device can be mapped to a LUN foreach client group of which it is a member.

A LUN for the created client group is then advertised to the client overthe FC network at operation 806. The advertised LUN can then bediscovered by the client so that the client can send SCSI requests tothat LUN. The server can advertise the LUN at one or more ports of oneor more server HBAs, according to the requirements of the client group.In some embodiments, multiple LUNs are created and advertised for eachclient group.

With the client added to a group and one or more LUNs advertised to theclient, the server is operable to communicate with the client using SCSIrequests and SCSI responses over the FC network. Accordingly, atoperation 807 the server receives a SCSI request from the client overthe FC network. The server can then service the SCSI request, such as byrouting the SCSI request to a server FC adapter. Where the server hasresponded to the SCSI request from the client, the server is operable tosend the SCSI response to the client over the FC network, as shown atoperation 808.

FIG. 9 illustrates an embodiment of a method 900 executed by a serverfor servicing SCSI requests received over a FC network from a client. Inone embodiment, the method 900 is performed where the server receives aSCSI request for a LUN that is routed to a server FC adapter. The method900 can be performed by the server FC adapter 320 of the server 150illustrated in FIG. 3.

Beginning at operation 901, a SCSI request originating at a client isreceived. This SCSI request can be a SCSI request for a LUN mapped to aSCSI device created by or associated with a server FC adapter of theserver. The SCSI request can be received from a server OS SCSI serviceoperable to receive SCSI requests over the FC network.

At operation 902, the server determines the type of request included inthe SCSI request. For example, the type of request can be a request toestablish a virtual connection, a request to send or receive a messagefrom the server, or a request to get information associated with theserver. For some SCSI requests, the SCSI request includes the requesttype in the LBA field of the SCSI request, such as a predetermined valueor other indicator. The request type can also be included in a headerwithin a payload of the SCSI request. For other SCSI requests, theserver is operable to identify a virtual connection identifier includedin the SCSI request and resolve the request type based on the virtualconnection identifier. In one embodiment, the server determines therequest type using the virtual connection identifier in combination withone or more other parameters, such as an SCSI operation code of the SCSIrequest (e.g., SCSI read) or an additional value in the LBA field orpayload of the SCSI request.

In response to determining the request type, the server determines howto handle the SCSI request, as shown at decision block 903. In oneembodiment, the SCSI request can include one of three request types: (1)a request to get information about the server, (2) a request establish avirtual connection, and (3) a request for an existing virtualconnection. A request to establish a virtual connection may include SCSIrequests for the assignment of a virtual connection identifier and otherrelated requests from the client to establish a virtual connection, suchas a SCSI request including a process descriptor. An embodiment of amethod for establishing a virtual connection is illustrated at FIG. 10.A request for an established virtual connection may include SCSIrequests including call messages or other data and SCSI requests toretrieve reply messages or other data. An embodiment of a method forcommunicating messages by a server is illustrated at FIG. 11.

A request to get server information may be received at the server wherethe server has not yet established a virtual connection with the client,or where the client is attempting to confirm or catalog informationabout the server. Where the SCSI request is a request to get serverinformation, the server responds with a SCSI response that includesinformation about the server at operation 904. In one embodiment, theserver creates a SCSI response that includes information about theserver included in a payload of the SCSI response. The serverinformation can include, for example, a server identifier, a serialnumber, and/or the transport path which the SCSI request traversed inreaching the server (e.g., a transport path including a physicalcomponent and a logical component). The request is then sent by theserver to the client over the FC network.

FIG. 10 illustrates an embodiment of a method 1000 for assigning avirtual connection to a client by a server connected with the clientover a FC network. As described above, the method 1000 is performedwhere the server receives SCSI one or more SCSI requests indicating theclient is attempting to establish a virtual connection. The method 1000can be performed in response to receiving a SCSI request from the clientat a server FC adapter of the server, such as a SCSI request routed tothe server FC adapter by a server OS SCSI service. The method can beperformed by the server FC adapter 320 of the server 150, illustrated inFIG. 3. With respect to the embodiment of FIG. 9, the received SCSIrequests illustrated in the method 1000 can be the SCSI requestsreceived at operation 901 and determined to be SCSI requests forestablishing a virtual connection at operations 902-903.

Beginning with operation 1001, a first SCSI request originating at theclient is received by the server over the FC network. The first SCSIrequest can indicate that the client is attempting to start theestablishment of a virtual connection. The SCSI request can be, forexample, a SCSI read request that includes an indication or other apredetermined value included in the LBA of the SCSI request.

In response to the received SCSI request, the server assigns a virtualconnection identifier to the virtual connection, as shown at operation1002. The virtual connection identifier can be, for example, a value andcan be part of a virtual connection tuple that includes a generationnumber and/or verifier assigned by the server to ensure that the virtualconnection can be uniquely identified across space and time. The servercan assign the virtual connection identifier by for example, generatinga value or selecting a value from a pool of available values.

At operation 1003, the server responds to the first SCSI request withthe virtual connection identifier. In one embodiment, the server createsa SCSI response that includes the virtual connection identifier and/orother parameters, such as the virtual connection tuple and/or a serveridentifier. The server can then send the first SCSI response to theclient over the FC network. In some embodiments, the server places thevirtual connection in a “waiting” state with a timeout. Where the serverdoes not receive a second SCSI request from the client for the virtualconnection before the timeout expires, the server can release thevirtual connection identifier and any resources associated with thevirtual connection.

After responding to the first SCSI request with the assigned virtualconnection identifier, the server receives a second SCSI request fromthe client over the FC network, as shown at operation 1004. The secondSCSI request includes a descriptor for a server process with which theclient is attempting to communicate. The second SCSI request can be aSCSI write request. In one embodiment, the second SCSI request includesthe assigned virtual connection identifier in the LBA field of thesecond SCSI request. The second SCSI request can also include parametersfor validation and creation of the virtual connection, such as thevirtual connection tuple. The parameters can be included in a payload ofthe second SCSI request. In one embodiment of operation 1004, the secondSCSI request includes a catalog of transport paths between the serverand the client over the FC network.

At operation 1005, the server associates the virtual connection with aserver process corresponding to the process descriptor included in thesecond SCSI request. The server can associate the virtual connection by,for example, mapping the virtual connection to the server process. Insome embodiments, operation 1005 occurs after some of the succeedingoperations of the method 1000. For example, the server can associate thevirtual connection at any point after the process descriptor is receivedat operation 1004. With the virtual connection associated, the server isoperable to receive and respond to messages adapted to SCSI requestsover the FC network. An embodiment of this communication process isillustrated at FIG. 11.

At operation 1006, the server responds to the second SCSI request with asecond SCSI response. This second SCSI response can indicate that theserver is able to establish the virtual connection. In one embodiment,this second SCSI response is contingent upon operation 1005. Forexample, the server FC adapter creates a second SCSI response indicatingthat the server is able to establish the virtual connection only wherethe server FC adapter first associates the virtual connection with theserver process. In one embodiment, the second SCSI response is a statuscode, such as a SCSI status code or vendor-specific status code. Theserver can then send the second SCSI response to the client over the FCnetwork. In some embodiments, the server again places the virtualconnection in a “waiting” state with a timeout. Where the server doesnot receive a third SCSI request from the client for the virtualconnection before the timeout expires, the server can release thevirtual connection identifier and any resources associated with thevirtual connection, and/or disassociate the server process.

At operation 1007, the server receives a third SCSI request over the FCnetwork that is to complete the establishment of the virtual connection.This third SCSI request can be a SCSI read request. The third SCSIrequest includes the virtual connection identifier, such as in the LBAfield of the third SCSI request.

The server can select a transport path to complete the establishment ofthe virtual connection. As shown at operation 1008, the server selectsthe transport path for the virtual connection from the catalog oftransport paths provided to the server at operation 1004. The catalog oftransport paths can be used by the server for load balancing and otheroptimization. The server can select a transport path so that SCSIrequests are more evenly distributed across ports, server HBAs, and/orLUNs of the server.

At operation 1009, the server responds to the third SCSI request withthe selected path for the virtual connection. In one embodiment, theserver creates a SCSI response that includes the virtual connectionidentifier and the transport path in a payload of the third SCSIresponse. The payload of the third SCSI response can include otherparameters, such as a virtual connection tuple. In response to the thirdSCSI request, the server can then send the third SCSI response to theclient over the FC network. In some embodiments, responding to the thirdSCSI request with the third SCSI response completes the establishment ofthe virtual connection at the server.

Turning to FIG. 11, a method 1100 illustrates an embodiment of a methodat a server for communicating messages between the server and a clientover a Fibre Channel network using SCSI requests and responses. Thismethod 1100 can be performed by the server FC adapter 320 operable tocommunicate with one or more processes 315 a-315 b of a server 150, asshown in FIG. 3. In some embodiments, the method 1100 is performed wherea virtual connection has been established with a client over the FCnetwork. Accordingly, SCSI requests received at the server from theclient include a virtual connection identifier for the virtualconnection. With respect to the embodiment of FIG. 9, the received SCSIrequest illustrated in the method 1100 can be a SCSI request received atoperation 901 and determined to be a SCSI request for an establishedvirtual connection at operations 902-903.

Beginning with operation 1101, a SCSI request originating at the clientis received by the server over the FC network. The received SCSI requestis for an established virtual connection and, accordingly, the SCSIrequest includes a virtual connection identifier. Additionally, the SCSIrequest can indicate that the client is attempting to send a callmessage to a server process or retrieve a reply message from a serverprocess. The SCSI request can be a SCSI read request that includes avirtual connection identifier in the LBA field of the SCSI request(i.e., a reply SCSI request). Alternatively, the SCSI request can be aSCSI write request that includes the virtual connection identifier and acall message (i.e., a call SCSI request).

At operation 1102, the server identifies a server process that isassociated with the virtual connection of the received SCSI request. Forexample, the server can maintain a map of the virtual connection to theserver process. Subsequently, the server can refer to the map toidentify the process using the virtual connection identifier included inthe received SCSI request. In one embodiment of operation 1102, theassociated server process is identified from a map of the virtualconnection to a socket identifier (e.g., a file descriptor) for anestablished socket connection between the server FC adapter and theserver process.

Following the identification of the associated process, the serverhandles the received SCSI message according to the type of request, asshown at decision block 1103. For some SCSI requests, the server isoperable to identify a virtual connection identifier included in theSCSI request and resolve the request type based on the virtualconnection identifier. In one embodiment, the server determines therequest type using the virtual connection identifier in combination withone or more other parameters, such as an operation code included in theCDB of the SCSI request (e.g., a SCSI read operation code) or anadditional value in the LBA or payload of the SCSI request. In someembodiments, the request type is determined at operations 902-903 ofFIG. 9. Consequently, the type of request can be resolved before theassociated server process is identified.

Where the received SCSI request is a call SCSI request (e.g., a SCSIwrite request including the call message in the payload), the method1100 proceeds to operation 1104. At operation 1104, the server extractsa call message from the SCSI request. The extraction can include, forexample, separating the call message from SCSI-specific or FC-specificdata. In some embodiments, this operation involves recognizing a headerin the call SCSI request and separating the header from the callmessage. For example, the header can include a number of bytes of thecall message in the payload and a padding or offset of the call messagebytes within the payload of the call SCSI request so that the callmessage bytes can be extracted.

In one embodiment, the call SCSI request includes an indication in theheader that a prior SCSI response sent by the server was received by theclient. The indication can be, for example, a sequence number includedin a prior reply SCSI response received by the client from the server.In response to the indication that the client received a prior SCSIresponse from the server, the server can increment the expected replysequence number and free resources consumed by the prior SCSI response,such as by removing a reply message included in the prior SCSI responsefrom one or more buffers.

With the call message extracted at the server, the call message isprovided to the process associated with the virtual connection atoperation 1105. The call message can be sent to the process or madeavailable so that the process can retrieve the call message, such as byreading the call message. In some embodiments, the virtual connectionincludes a connected stream socket between a server FC adapter and theassociated process. The server FC adapter can therefore write to thesocket opened for the associated process. Alternatively, the server FCadapter can provide the call message so that it can be read from thesocket connected with the associated process.

At operation 1106, the server responds to the received SCSI request witha SCSI response. This SCSI response can indicate that the server is ableto accept the entire call message included in the SCSI request. In oneembodiment, the SCSI response is a status code, such as a SCSI statuscode or vendor-specific status code. The server can then send the SCSIresponse to the client over the FC network.

In one embodiment, this SCSI response is contingent upon operation 1105.For example, the server FC adapter creates a SCSI response indicatingthat the server is able to accept the entire call message only where theserver FC adapter first buffers the entire call message or provides theentire call message to the associated process. In some instances, theserver is unable to accept the entire call message. Therefore, theserver can create a SCSI response indicating that some or none of thecall message is accepted. In some embodiments, a SCSI responseindicating that the entire call message is not accepted can include anumber of bytes of the call message that are accepted by the server.

Where the received SCSI request is a reply SCSI request (e.g., a SCSIread request for an established virtual connection), the method 1100proceeds to operation 1107. At operation 1107, the server receives areply message from the associated process. The reply message may be areply to a call message, such as data for a call message to get thatdata. The reply message can be sent by the associated process to theserver FC adapter or made available so that the server FC adapter canread the reply message. In some embodiments, the virtual connectionassociated with the identified process includes a stream socketconnection between the server FC adapter and the associated process. Theserver FC adapter can therefore read the reply message from the socketconnected with the identified process. Alternatively, the associatedprocess writes the reply message to the socket connected with thevirtual connection.

At operation 1108, the server responds to the received SCSI request witha reply SCSI response. In one embodiment, the server creates the replySCSI response so that a payload of the response includes the virtualconnection identifier and the reply message. The payload of the replySCSI response can include other parameters, such as a virtual connectiontuple. In some embodiments, the reply message is incomplete. Forexample, the reply message may be responsive to an earlier-received callmessage requesting data, but the reply message may only contain aportion of the requested data. The reply SCSI response can indicate thatthe reply message does not contain all of the requested data, such as byincluding a number of bytes returned in the reply message and/or anumber of bytes that are available at the server in response to a callmessage but not returned in the instant reply SCSI response. The servercan then send the reply SCSI response to the client over the FC network.

In one embodiment, the reply SCSI response includes an indication thatthe server requests that the client migrate to a different transportpath for future SCSI requests. This indication can be, for example, aflag or Boolean value, and/or the presence of a different transport pathin a header of the reply SCSI response. The different path may beselected from a catalog of transport paths received by the server froman earlier SCSI request from the client. In this way, the server canbalance the load of received SCSI requests across the server HBAs and/orLUNs.

Turning now to FIG. 12, the method 1200 illustrates an embodiment of amethod executed by a server for servicing SCSI requests received over aFC network from a client. In one embodiment, the method 1200 can beperformed by the server FC adapter 320 of the server 150 illustrated inFIG. 3. The method 1200 can be performed in response to receiving a SCSIrequest from the client at a server FC adapter of the server, such as aSCSI request routed to the server FC adapter by a server OS SCSIservice. With respect to the embodiments of FIGS. 9-11, the method 1200is not mutually exclusive, and some operations of FIGS. 9-11 can beperformed in addition to those operations illustrated in the method1200. For example, operations 901-903 may still be performed todetermine the request type. In such an example, the method 1200 isperformed for both virtual connection assignment requests and requestsfor an established virtual connection.

Beginning with operation 1201, the server receives a first SCSI requestfrom the client over the FC network. The first SCSI request includes adescriptor for a server process with which the client is attempting tocommunicate. Additionally, the first SCSI request includes a virtualconnection identifier for a virtual connection. This virtual connectionidentifier may have been assigned by the server, such as described atoperations 1001 and 1002 of FIG. 10. In one embodiment, this operation1201 is analogous to operation 1004.

At operation 1202, a socket is created and connected to a server processusing the process descriptor. A server FC adapter can create and connectthe socket. By creating the socket, a socket identifier, such as a filedescriptor, is returned. Thus, the socket identifier is received by theserver FC adapter. In one embodiment, the process descriptor is a portnumber. The server FC adapter can establish a connection with the serverprocess by connecting the created socket to the server process using theport number. The established connection can be a localhost connection ora remote connection. The server FC adapter can then write to the socket,poll the socket and read from the socket to communicate messages (e.g.,data) to and from the server process.

Once the socket is created and connected to the server process, thesocket can be associated with the virtual connection identifier, asillustrated at operation 1203. The server can associate the socket withthe virtual connection by, for example, mapping the virtual connectionidentifier to the socket identifier. The operations 1202-1203 can be anembodiment of operation 1005. Accordingly, the virtual connection can beestablished following operation 1203—for example, operations 1006-1009can be performed.

In one embodiment of operation 1203, two or more threads are created andattached to the socket so that messages can be continuously written toand read from the socket. For example, the server FC adapter can writecall messages to one or more buffers and attach those buffers to thevirtual connection. A write thread then asynchronously writes thebuffered call messages to the socket associated with the virtualconnection. A read thread can then poll the socket and read replymessages into one or more buffers, which the second thread then attachesto the virtual connection. The server FC adapter can receive the replymessages from the buffers attached to the virtual connection.

At operation 1204, a second SCSI request originating at the client isreceived by the server over the FC network. The received SCSI request isfor the established virtual connection and, accordingly, the SCSIrequest includes a virtual connection identifier. Here, the second SCSIrequest can indicate that the client is attempting to send a callmessage to a server process (i.e., a call SCSI request). In someembodiments, this operation 1204 is analogous to operation 1101 of FIG.11. Accordingly, operation 1102 follows operation 1204 in someembodiments of the method 1200. In one embodiment, the server processcan be identified by the socket connected to the process.

Proceeding to operation 1205, a call message is extracted from the SCSIrequest. In one embodiment, the extraction is analogous to operation1104. The extracted call message can then be written to the sockethaving the socket identifier associated with the virtual connectionidentifier included in the second SCSI request, as shown at operation1206. This operation 1206 can include writing the call message to one ormore buffers and attaching the one or more buffers to the virtualconnection. A write thread can then write the buffered call message tothe socket. Operation 1206 illustrates one embodiment of operation 1105.Therefore, operation 1106 can follow operation 1206 in some embodimentsof the method 1200.

At operation 1207, a reply message is read from the socket associatedwith the virtual connection. In some embodiments, the server FC adapteris polling the socket and, where the reply message is available, theserver FC adapter reads the reply message from the socket. Operation1207 can include a read thread that polls the socket and reads theavailable reply message into one or more buffers, which are thenattached to the virtual connection. Depending upon the available bufferspace, the read thread can read an entire reply message to one or morebuffers or a portion of a reply message. Operation 1207 illustrates oneembodiment of operation 1107.

Continuing to operation 1208, a third SCSI request originating at theclient is received by the server over the FC network. The third SCSIrequest is for an established virtual connection and, accordingly, thethird SCSI request includes a virtual connection identifier. The thirdSCSI request can be a SCSI read request that includes a virtualconnection identifier in the LBA of the SCSI request (i.e., a reply SCSIrequest). In one embodiment, the third SCSI request is identified as areply SCSI request according to an embodiment of operation 1103; forexample, the server FC adapter can determine that the third SCSI requestis a reply SCSI request by examining the LBA for the virtual connectionidentifier and the operation code for the SCSI-read operation code.

At the operation 1209, the third SCSI request is responded to with areply SCSI response. In one embodiment, the server creates the replySCSI response that includes the virtual connection identifier and thereply message in the payload of the reply SCSI response. The payload ofthe reply SCSI response can include other parameters, such as a virtualconnection tuple. In some embodiments, the reply message is incomplete.For example, a read thread may only be capable of buffering a portion ofa reply message available at the socket. The reply SCSI response canindicate that the payload of the reply SCSI response does not containthe full reply message. An embodiment of operation 1209 is described atoperation 1108. The server can then send the reply SCSI response to theclient over the FC network. In one embodiment, the reply SCSI responseincludes an indication that the server requests that the client migrateto a different transport path for future SCSI requests, as describedwith respect to operation 1108.

Turning to FIG. 13, a method 1300 illustrates an embodiment of a methodexecuted by a client for communicating messages between the client and aserver over a Fibre Channel network using SCSI requests and responses.This method 1300 can be performed by a FC transport adapter 214communicatively coupled with one or both of a data optimization module212 and a data storage module 213 of a client 110, as shown in FIG. 2.In some embodiments, the method 1300 is performed where a virtualconnection has been established with a registered server over the FCnetwork. Accordingly, the SCSI requests include a virtual connectionidentifier associated with a client module. With respect to theembodiments of FIGS. 6-7, the method 1300 is not mutually exclusive, andsome operations of FIGS. 6-7 can be performed in addition to thoseoperations illustrated in the method 1300. For example, operation 701may still be performed to receive a call message from a client module.The method 1300 can be included in FIGS. 6-7 to provide reliablecommunication of messages.

Beginning with operation 1301, the client creates a first SCSI request,such as a call SCSI request or a reply SCSI request. The virtualconnection identifier is included in this first SCSI request, such as inthe LBA field. Other parameters can be included in the LBA field, suchas a timeout value and/or a sequence number (or a bit segment of thesequence number). Additionally, a SCSI write request can include otherparameters as part of its payload. For a call SCSI request, therequest's payload can include a call message byte number and/or thenumber of bytes of the call message included in the payload. Twoembodiments of operation 1301 are described at operations 702 and 704 ofFIG. 7.

Subsequently, the client sends the first SCSI request to a server overthe FC network, as shown at operation 1302. The client can send the SCSIrequest to the server over the FC network using any transport path tothe registered server, such as a transport path received from the serverduring the establishment of the virtual connection.

At operation 1303, the client determines a message status of the firstSCSI request. In one embodiment, the message status is one of fourcategories: (1) complete, (2) incomplete, (3) invalid, and (4) failed.The client can determine that the message status of the first SCSIrequest is complete where all of a message has been communicated betweenthe client and the server. For example, the client can determine thatthe message status of a call SCSI request is complete where the serverhas accepted the entire call message. Alternatively, the client candetermine that the message status of a reply SCSI request is completewhere the client has received the entire reply message from the server.

In some instances, the server is unable to completely accept a callmessage (e.g., the server has insufficient buffer space for the entirecall message) or completely send a reply message (e.g., the replymessage is not available for communication at the server or is too largeto include in a payload of a single SCSI response). Consequently, theclient can receive a first SCSI response from the server over the FCnetwork that indicates this incompletion. The SCSI response can include,for example, a status code and/or an indication of a number of bytesaccepted for a call message or the number of bytes returned for replymessages. The client then can determine that the message status of thefirst SCSI request is incomplete.

To maintain data integrity, the first SCSI request sent by the client tothe server over the FC network is validated at the server. In oneembodiment, the first SCSI request includes the virtual connectionidentifier in the LBA field of the first SCSI request. The server canvalidate the first SCSI request using the virtual connection identifierand/or other parameters in the LBA, such as a sequence number.Additionally, where the first SCSI request is a call SCSI request havinga payload, the server can validate the SCSI request using parametersincluded in the payload, such as a virtual connection tuple. If theserver determines that the first SCSI request is invalid, the clientreceives a SCSI response from the server over the FC network indicatingthe first SCSI request is invalid. The client can determine that themessage status is invalid upon receiving such a SCSI response from theserver.

Additionally, a SCSI response for the first SCSI request can bevalidated at the client. For example, the client can validate a SCSIresponse using a header of the response that includes the virtualconnection identifier and/or other parameters included in the header,such as a virtual connection tuple and a sequence number. If the clientdetermines that the SCSI response is invalid, the client can determinethat the message status is invalid.

Occasionally, the first SCSI request or the first SCSI response fails tobe communicated, such as due to a failure of software or hardware at theclient, the server or the FC network. The client can determine thefailed message status where a client timeout expires and a SCSI responsehas not been received. Alternatively, the client can determine thefailed message status of the first SCSI request by receiving anotification that the SCSI request failed (e.g., from a client OS SCSIservice) or by receiving a SCSI response from the server indicating thatthe first SCSI request was aborted (e.g., before reaching a server FCadapter).

Where the message status is determined to be complete, normal messagecommunication using SCSI requests and responses over a FC network canresume, as shown at decision block 1304. An embodiment of this processis illustrated at FIG. 7, and the process can resume at, for example,operations 701, 702 or 704.

If the message status is not complete, the client can determine anaction based on the determined message status, as shown at operation1305, such as retrying the SCSI request or closing the virtualconnection. In one embodiment, the client can provide the status to themodule, such as by indicating the call or reply message could not besent or received or by indicating a socket failure at the FC transportadapter. In response, the module can instruct the FC transport adapterto end message communication, such as by closing a socket connectionbetween the module and the FC transport adapter. Thus, FC transportadapter can disassociate the module and the virtual connection and ceasecreating SCSI requests for message communication for that virtualconnection.

In an embodiment in which the client determines the message status isincomplete, the client can determine that the action is to complete themessage communication by creating a next SCSI request at operation 1306.Where the first SCSI request includes a call message, the client cancreate a next call SCSI request that includes the remainder of the callmessage that was not accepted by the server. Where the first SCSIrequest is a reply SCSI request, the client can create a next reply SCSIrequest that requests the remainder of the reply message.

In an embodiment in which the client determines the status is invalid,the client determines that the action is to close the virtualconnection. In one embodiment, the client closes a virtual connection bysending a SCSI request to the server over the FC network to request thatthe virtual connection be closed. Thus, at operation 1306 the clientcreates a next SCSI request (e.g., a SCSI write request) that includesthe virtual connection identifier and an indication that the virtualconnection is to be closed at the server. The client can alsodisassociate the virtual connection from the associated module.

In an embodiment in which the client determines the message status isfailed, the client can determine that the action is to retry the firstSCSI request. Accordingly, at operation 1306 the client can create anext SCSI request that is substantially the same as the first SCSIrequest. In some embodiments, the next SCSI request does not incrementthe message sequence number but uses the same sequence number from thefirst SCSI request, because the client assumes that the first SCSIrequest did not reach the server and therefore the expected sequencenumber at the server would not have been incremented. Additionally, theclient can select a new transport path for the next SCSI request toaddress a failure.

At operation 1307, the client sends the next SCSI request to a serverover the FC network. The method 1300 then returns to operation 1303 anditerates through the method 1300 as described above.

Now with respect to FIG. 14, a method 1400 illustrates an embodiment ofa method executed by a server for reliably communicating messagesbetween the server and a client over a Fibre Channel network using SCSIrequests and responses. This method 1400 can be performed by a server FCadapter 320 operable to communicate with one or more processes 315 a-315b of the server 150, as shown in FIG. 3. In some embodiments, the method1400 is performed where a virtual connection has been established with aclient over the FC network. Accordingly, the SCSI requests include avirtual connection identifier associated with a server process. Withrespect to the embodiments of FIGS. 11-12, the method 1400 is notmutually exclusive, and some operations of FIGS. 11-12 can be performedin addition to those operations illustrated in the method 1400. Forexample, the operations 1201-1203 of FIG. 12 may be performed to createa socket connection to a server process and associate the socket with avirtual connection. The method 1400 can be included in FIGS. 9-12 toprovide reliable communication of messages.

Beginning with operation 1401, a first SCSI request is received at theserver. The received SCSI request is for an established virtualconnection and, accordingly, the SCSI request includes a virtualconnection identifier. The SCSI request can be a SCSI read request thatincludes a virtual connection identifier in the LBA field of the SCSIrequest (i.e., a reply SCSI request). Alternatively, the SCSI requestcan be a SCSI write request that includes the virtual connectionidentifier in the LBA field and a call message, and/or a header, in thepayload (i.e., a call SCSI request). Embodiments of operation 1401 aredescribed at operations 1101, 1204 and 1208.

At operation 1402, the first SCSI request is validated. The first SCSIrequest can be validated by examining the virtual connection identifier,such as by comparing it to an expected virtual connection identifier.Additionally, the first SCSI request can be validated by examining otherparameters included therein. In one embodiment, the first SCSI requestincludes a tuple value of a virtual connection tuple (or a bit segmentthereof) and the included tuple value is compared to an expected tuplevalue for that virtual connection. The server can examine the virtualconnection identifier and other parameters at the LBA field of the firstSCSI request. For a SCSI request that includes a payload (e.g., a callSCSI request), the server can use other parameters included in a headerof the payload to validate the first SCSI request, such as the virtualconnection tuple, in addition to or instead of the parameters in the LBAfield. Where the server encounters an unexpected virtual connectionidentifier, the server can determine that the first request is invalid.In one embodiment, validation is contingent upon one or more parametersincluded in the first SCSI request, in addition to the virtualconnection identifier.

In one embodiment, the first SCSI request includes a sequence number inaddition to the virtual connection identifier. The server can comparethe sequence number to an expected sequence number. Where the twosequences numbers do not match, the server can determine that therequest is invalid. However, where the server determines that therequest is otherwise valid and the received sequence number matches alast expected sequence number, the server can assume that the client didnot receive a last SCSI response sent by the server over the FC network,and therefore the client is retrying the SCSI request. A retried requestcan be considered either valid or invalid, depending upon theembodiment.

Where the first SCSI request is invalid, the server proceeds tooperation 1405. At operation 1405, the server responds to the first SCSIrequest with a first SCSI response indicating that the first SCSIrequest is invalid. In one embodiment, the server FC adapter creates afirst SCSI response that includes a SCSI status code or vendor-specificstatus code to indicate the invalidity. The first SCSI response is thensent to the client over the FC network.

Where the first SCSI request is valid, the server continues to operation1403 to determine a message status of the first SCSI request. Themessage status can be based on attempting to communicate a message witha server process associated with the virtual connection identified inthe first SCSI request (e.g., send a call message to or receive a replymessage from the process).

Where the first SCSI request is a call SCSI request, the message statuscan be determined based on whether the server accepts the entire callmessage. Two embodiments of this are illustrated at operations 1104-1105and 1205-1206. For example, a server FC adapter can accept the entirecall message by providing the call message to a server processassociated with the virtual connection or by buffering the call messageto be provided to the associated process and, therefore, the messagestatus is complete. Where the server only accepts part of the callmessage, the message status is incomplete.

Similarly, where the first SCSI request is a reply SCSI request, themessage status can be determined based on whether a reply message isavailable for the virtual connection of the reply SCSI request. Twoembodiments of this are illustrated at operations 1107 and 1206. Forexample, a server FC adapter receive the entire reply message from aserver process associated with the virtual connection or by receivingthe entire message from one or more buffers and, therefore, the messagestatus is complete. Where the server only a portion of or none of thereply message is available, the message status is incomplete.

In some embodiments, the message communication is constrained by atimeout. If the server is unable to accept the call message or if anentire reply message is unavailable before the timeout expires, themessage status can indicate incomplete.

In one embodiment, the first SCSI request includes an indication that aprior SCSI response sent by the server was received by the client. Theindication can be, for example, a sequence number of a prior reply SCSIresponse received by the client from the server. In response to theindication that the client received a prior SCSI response from theserver, the server can free resources consumed by the prior SCSIresponse, such as by removing a reply message included in the prior SCSIresponse from one or more buffers. Additionally, the client canincrement the expected reply sequence number.

At operation 1404, the first SCSI request is responded to with a firstSCSI response based on the message status. For example, if the server isable to accept or provide an entire call or reply message, the servercan create a first SCSI response indicating that the message status iscomplete. Where the server is unable to accept or provide an entire callor reply message, the server can create a SCSI response indicating thatthe message status is incomplete. In some embodiments, the SCSI responsecan include a number of bytes of an incomplete call message that wereaccepted or not accepted by the server. Alternatively, the SCSI responsecan include a number of bytes of an incomplete reply message and/or anindication that the reply message is incomplete (e.g., the number ofbytes requested and the number of bytes actually included do not match).Three embodiments of this operation are described at operations 1106 and1108 of FIG. 11 and operation 1209 of FIG. 12. The server can then sendthe SCSI response to the client over the FC network.

In embodiments wherein a retried SCSI request is not considered invalid,the server can respond with the prior SCSI response. The server can havethe prior SCSI response buffered or cached so that a retried SCSIrequest can be quickly responded to by the server without consumingadditional resources. The server can then send the SCSI response to theclient over the FC network.

At the end of the method 1400, normal message communication using SCSIrequests and responses over a FC network can resume. An embodiment ofthis process is illustrated at FIG. 9.

FIG. 15 is a flowchart illustrating one embodiment of a method executedby a server for selecting paths for virtual connections. In oneembodiment, the method begins with the operation 1501 where the serverreceives a request to establish a connection that is serviced by avirtual connection over the Fibre Channel network using SCSI messagessent by the client. As detailed further herein above, the operation 1503continues with the server Fibre Channel adapter receiving a set orcatalog of available paths over the Fibre Channel network between theclient and the server, or more specifically the resource (e.g., a serverhost bus adapter, client host bus adapter, and a LUN) that the client isseeking to communicate with.

At operation 1505, the server Fibre Channel adapter receives loadconditions for endpoints of each path. The load conditions can bemeasured by a separate monitoring module or similar component of thesystem. The load can be measured in throughput, resource usage, queuelength or similar metrics. The load is measured on an endpoint byendpoint basis. The local endpoints can be monitored by a module of theserver Fibre Channel adapter at the server and/or a module of the FibreChannel transport adapter at the client. The server and client canexchange this data using the SCSI over Fibre Channel protocols asdescribed herein or using other methods of communication.

The operation 1507 then selects the path with the lowest load at itsendpoints. Where endpoint load is known for both ends the totalcumulative load can be considered when selecting the path, theserver-side endpoint can be given primary consideration or weight withthe client-side load being a secondary or tie-breaking consideration orweight. Where endpoint load is only known for the server-side endpointsthen the path having the lowest load at the server-side endpoint can beselected. In some embodiments, the LUNs of a path are considered as atertiary component of the load. Where the load across the LUNs of aselected server-side endpoint are unevenly distributed, then a less-busyLUN of that server-side endpoint can be selected for the path. Theselected path is then assigned to the virtual connection at operation1509. The server Fibre Channel adapter can return the selected path thatis assigned to the virtual connection as described herein above.

FIG. 16 is a flowchart illustrating one embodiment of a method executedby a server for rebalancing virtual connections over available paths.This process can be executed by a virtual connection rebalancing modulethat is part of or in conjunction with the server Fibre Channel adapter.A rebalancing of the distribution of virtual connections over the pathsand endpoints can be analyzed intermittently at defined intervals or inresponse to heavy load conditions at particular endpoints. In oneexample embodiment, at operation 1601 the rebalancing is initiated at adefined interval. At operation 1603 the current load conditions for thevirtual connections are checked. The check of the virtual connectionscan determine the load at the endpoints associated with each virtualconnection as well as the overall virtual connection load. Themonitoring of the load can be on the server-side by a local monitoringmodule or can be at both the server-side and the client-side where theclient executes a monitoring module to collect load information for theclient-side endpoints of the paths associated with the virtualconnections. The load can be measured in throughput, resource usage,queue length or similar metrics.

At operation 1605, the load on a particular path assigned to a virtualconnection can be determined to exceed a define threshold. If such athreshold were not exceeded, then the process would continue at the nextinterval at operation 1601. The operation 1607 checks the load onalternate paths for a virtual connection, in response to determiningthat the load on the path of the virtual connection has exceeded thethreshold. The alternate paths can be known from the catalog or set ofpaths that was provided by the client at the time of the selection ofthe initial path for the virtual connection or by a recalculation of theavailable paths based on current topology data maintained by the server.The alternate path having the lowest or minimum load is selected by thevirtual connection rebalancing module at operation 1609.

The server Fibre Channel adapter at operation 1611 then migrates thevirtual connection to the selected alternate path. The virtualconnection can be updated with the path identifier or path information.At operation 1613, this path identifier or path information is sent tothe client via a SCSI message to direct the client to utilize theselected alternate path over the Fibre Channel network for the specifiedvirtual connection. This process can continue intermittently or at thedefined interval to continuously check and rebalance the distribution ofthe virtual channels over the paths between a set of clients and theserver and thereby optimize the use of resources and throughput for theserver.

FIG. 17 is a block diagram of one embodiment of a client-server systemfor reliable communication over a Fibre Channel network. Theclient-server system is described in further detail herein above inregard to FIG. 4A. The virtual connection balancing module 1701 is shownhere as being a component of the server Fibre Channel adapter where itmonitors the load of the virtual connections and dynamically reassignsthem to less loaded endpoints or paths between the server and the clientcommunicating with the server via the virtual connection. In thismanner, the virtual connection balancing module improves the throughputand reliability of the SCSI over Fibre Channel communication system.

FIG. 18 is a flowchart illustrating one embodiment of virtual connectionengine instantiation. The method is executed by the server at time thatthe server is started up or the start of the services provided by thevarious server processes are initiated such that the ability tocommunicate with the client processes over the Fibre Channel networkusing SCSI may be required. At operation 1801 an operating system orsimilar component of the server starts the execution of the server FibreChannel adapter. The server Fibre Channel adapter then determines theresources available at the server or a set of servers over which itoperates and facilitates communication between the server and a set ofclients and the processes or applications executing on the set ofclients.

At operation 1803, the server Fibre Channel adapter identifies a set oflocality domains. Locality domains are sets of processing units, such ascentral processing units, and the resources, such as memory, caches andnetwork bandwidth, that are associated and available to each of theprocessing units. These locality domains can be contained within adiscrete server machine or can be distributed over multiple machines orsimilarly arranged. In one embodiment, resources and processorsallocated to one locality domain cannot be allocated or shared to anyother locality domain. These locality domains can remain fixed duringthe operation of the server or in other embodiments can be dynamicallyrearranged as resources change or in response to failures within theserver system. The locality domains are conceptual units of operationthat are maintained by the server Fibre Channel adapter to manage theresources that are available to the server Fibre Channel adapter.

At operation 1805 at least one virtual connection engine is generatedand assigned to each of the locality domains. A virtual connectionengine is a collection of processing threads that handle the functionsof a set of virtual connections. In one example embodiment, thesethreads handle processing of incoming SCSI request including DATA_SENDand DATA_RECEIVE operations (referred to as an Engine Control Thread),the writing of buffered data to the backend local host sockets tied tothe processes of the server (referred to as a Data Send Poll Thread) andthe reading of data from the backend local host sockets into buffers tobe provided to client systems via SCSI response messages (referred to asa Data Receive Poll Thread). The virtual connection engine (VCE) canguarantee a single producer/consumer model for handling a set of virtualconnections. Production (i.e., adding data to the stream) is fullycontrolled by one thread and consumption (i.e., removing data from thestream) is fully controlled by one thread. The producer and consumerthreads are separate and independent. A VCE can handle any number ofvirtual connections, however, an uneven distribution of the virtualconnections can diminish performance. This performance can, for example,impact data cache utilization by the virtual connections. With a VCEsharing data cache resources amongst the virtual connections assigned tothe VCE, the data cache can become a bottleneck for the operations ofthe virtual connections assigned to a VCE with a heavy load.

FIG. 19 is a flowchart illustrating one embodiment of virtual connectiongeneration and load distribution. As described herein above, after beingestablished the server Fibre Channel adapter can establish virtualconnections in response to requests from clients over the Fibre Channelnetwork using SCSI messaging. This process can be executed by a VCEmanagement module of the server Fibre Channel network, which performsthe operations that generates and assigns virtual connections to VCEs.At operation 1901, the VCE management module receives a new connectionrequest from a client in the form of a SCSI message.

At operation 1903, the VCE management module generates a virtualconnection for the client to service the communication request betweenthe client and the server. At operation 1905, the VCE management moduledetermines a load for each of the VCEs in the server Fibre Channeladapter. The load can be determined by metrics such as throughput, queuelength, processing time, or similar metrics. The VCE management moduleselects the VCE with a minimum load at operation 1907. This provides aninitial load distribution upon creation of each virtual connection.However, this load balance can change over the operation of a set ofvirtual connections as some virtual connections generate a heavier loadover time.

FIG. 20 is a block diagram of one embodiment of a client-server systemfor reliable communication over a Fibre Channel network. The exampleclient-server system is introduced herein above in regard to FIG. 4A andthe additional components of a virtual connection balancing module 1701,VCE management module 2001 and local domains 2011. The exampleembodiment includes a single server and client by way of example. Oneskilled in the art would understand that any number of servers andclients can interact using the SCSI over Fibre Channel network and thatthe components described herein can be distributed over any number ofservers or clients.

The VCE managing module 2001 can generate virtual connections or takeresponsibility for assigning virtual connections 2007 to particular VCEs2005. New virtual connections 2007 can be assigned to any VCE 2005. Inone embodiment, the VCE managing module 2001 assigns new virtualconnections to a VCE with a minimum load to create an initial loadbalance amongst the VCEs. Each VCE can have access to a set of domainresources specific to the local domain 2011.

The VCE balancing module 1701 analyzes the load on each of the VCEs 2005to determine whether any VCE has an excessive load or a load thatexceeds a particular threshold. If such a VCE is found, then the VCEbalancing module 1701 reassigns a set of virtual connections from theVCE with the high load to another VCE such as a VCE with a minimum orlow load. The VCE balancing module 1701 can check the load balance atany interval or with any frequency and can check each VCE or a subset ofthe VCEs. The VCE balancing module 1701 can obtain the metrics frommonitoring modules or similar sources for determining the VCE andvirtual connection loads.

FIG. 21 is a flowchart illustrating one embodiment of a virtualconnection rebalancing process. This process is carried out by the VCEbalancing module. At operation 2101, the VCE balancing module starts therebalance of virtual connection (VC) assignments to each of the VCEs.This rebalancing method can take place with any frequency or with anyinterval. The VCE balancing module obtains the current VCE load for eachVCE as well as the load contributed to each VCE by each virtualconnection at operation 2103. The load data can be obtained from amonitoring module or similar source.

At operation 2105, the VCE rebalancing module checks whether the load oneach VCE exceeds a defined threshold load. The threshold can be set byan administrator, dynamically determined or pre-programmed. A check canbe made for each VCE or can be made just until at least one VCE is foundto exceed the threshold. If no VCEs have a load that exceeds thethreshold, then the method continues by waiting until the nextrebalancing iteration at operation 2101.

However, if at least one VCE is found to exceed the threshold, then atoperation 2107 the VCE rebalancing module reassigns at least one virtualconnection of the VCE with the highest load or the load that exceededthe threshold. The virtual connections that are reassigned can bereassigned to the VCE that has a minimum load or at least a VCE with aload below the threshold. In one embodiment, the virtual connectioncontributing the largest load to the VCE is the virtual connection thatis reassigned. In another embodiment, any set of virtual connectionsthat reduce the load of the VCE below the threshold, to an average loador similar standard can be reassigned to another VCE such that it doesnot as a result of the reassignment exceed the threshold.

FIG. 22 is a block diagram of one embodiment of shared access system formanaging data streams in virtual connections. This method and systemoptimize the utilization of the data resources utilized by each virtualconnection. Specifically, the latency that is caused by the buffering ofdata by the virtual connection to be read from or written to the localhost sockets of the local server process. The method minimizes the timerequired to lock data structures to ensure coherency, thereby reducinglatency because the producer thread and consumer thread can nearlycontinuously access the data streams to process the data streams. A datastream consists of a singly-linked list, where each item in the list isa buffer that can hold any amount of data, (e.g., 64 kb of data).

As discussed above, each VCE can have three threads that process thedata for all virtual connections that are serviced by the VCE. The threethreads are the Engine Control Thread, Data Send Poll Thread and theData Receive Poll Thread. With this method, the VCE can guarantee asingle producer and single consumer model of operation. The productionmethod of adding data to the data stream can be fully controlled by onethread and the consumption method of removing data from the data streamcan be controlled by one thread. The producer and consumer threads aredifferent for the two data streams associated with each virtualconnection. There is a send data stream for the data to be forwarded tothe local host socket for the server process. There is also a receivedata stream for the data received from the local host socket from theserver process.

For the send data stream the producer thread is the Engine ControlThread for most incoming data where the Send_Data operation can besatisfied quickly, because the send data stream is not full. In othersituations, the Data Send Poll Thread can be a producer for this datastream when handling a pending Send_Data operation after some data hasbeen removed from the send data stream and written to the local hostsocket. The consumer for the send data stream is always the Data SendPoll Thread. For this data stream, the producer thread seeks to be ableto add data to the data stream (e.g., implemented as a queue) while theData Send Poll Thread is seeking to remove data items from the send datastream and write them to the backend local host socket. Avoidingblocking the Engine Control Thread while the Data Send Poll is writingminimizes any latency associated with the send data stream.

For the receive data stream the producer is always the Data Receive PollThread. The consumer is typically the Engine Control Thread in the casewhere the Receive Data operation can be satisfied quickly out of dataalready present in the receive data stream. In other cases, the DataReceive Poll Thread may be a consumer, when handling a pending ReceiveData operation, after some data has been read out of the backend localhost socket into the receive data stream. The consumer thread seeks tobe able to remove data from the receive data stream (e.g., implementedas a queue) while the Data Receive Poll Thread filling the receive datastream via a read operation from the backend local host socket. It isdesirable to avoid blocking the Engine Control Thread while the DataReceive Poll Thread is performing a read.

Based on a single producer/single consumer model of operation as definedherein above the time locking each data stream is minimized therebymaximizing throughput. These structures are shown for an example set ofmessages being processed between the client and the server process bythe server Fibre Channel adapter. The structures shown are isolated forsake of clarity from the general structures shown for example in FIG. 4Aand discussed herein above.

The server Fibre Channel adapter 454 enables communication between aserver process and a client. In the example, the client sends a SCSIWrite message 2211 with a payload of data to be provided to the serverprocess. The virtual connection places this data in the send data stream2205, which includes a queue and state data to track the currentconditions of the queue including in one embodiment a lock. The datafrom the payload and the SCSI Write can be handled by the Engine ControlThread, or by the Data Send Poll Thread itself for pending operations,which stores the data in the tail of the send data stream queue. TheData Send Poll Thread then reads this data from the data stream when itreaches the head of the queue and writes it to the backend local hostsocket as a message 2201 for the server process.

The server process may generate a response message 2203 with data to bereturned to the client. This message and data are handled by the DataReceive Poll Thread, which stores the data in the receive data stream atthe tail of the queue. The Engine Control Thread retrieves this data inresponse to receiving the SCSI Read message 2213 from the client, whichgenerates a SCSI Read Response message 2209 with the data from the headof the receive data stream. The Data Send Poll Thread can also retrievethe data for pending operations. The processes of the producer andconsumer threads are further described in regard to FIG. 23 and FIG. 24.

Implementing this system and process, the producer and consumer cansimultaneously access an individual buffer in the linked list of thedata stream, without the need for locking or only a very brief andlimited use of a lock. This is because the producer is the only processthat adds a buffer to the linked list, and only adds a buffer when italready holds some data. Also, the consumer is the only process thatremoves a buffer from the linked list of the data stream and only afterit has consumed all data from every byte position within the buffer. Alock is held only when buffers are being added to or removed from thelist.

This simultaneous access of a buffer without locking provides aperformance advantage, because a producer and consumer can access thesame buffer without having to wait to obtain a lock, which reduces idletime where one or the other must wait for the lock.

FIG. 23 is a flowchart illustrating one embodiment of a consumer methodfor shared data stream management in a virtual connection. The consumerprocess is described in regard to the management of the data from thesend data stream. However, one skilled in the art would understand thatthe principles and operations of this process are applicable andadaptable to the management of the receive data stream as well. For sakeof clarity the example of the management of the consumer method as it isapplied to the send data stream is provided. The consumer thread can beinstantiated at the time that the VCE is created and allotted a scheduleto service the virtual connection containing the data streams.

The operation 2301 detects availability of the destination port (i.e.,the backend local host socket) association with the virtual connection,which is a mechanism for communication between a server process and aclient that utilizes the SCSI over a Fibre Channel network. In responseto detecting the availability of the port, the consumer thread checkwhether there is data available in the data stream (i.e., the queue ofthe send data stream in this example) of the virtual connection atoperation 2303. If there is no data in the data stream, then the methodcontinues and checks again in subsequent iterations whenever the threadis available to the virtual connection at operation 2301.

If there is data available in the data stream then the consumer threadreads the available data from the head of the queue, which in oneexample is implemented as a singly linked list at operation 2305. Thedata read from the head of the queue is written or forwarded to theavailable destination port or backend local host socket in route to theserver process at operation 2307. A check is then made whether all thedata from the head of the queue has been read and forwarded or similarlyconsumed by the consumer thread. If all of the data has not been read orconsumed, then the method continues allowing the consumer thread tocontinue to read and transfer data to the server process.

If however, the reading of the data has exhausted the available data inthe head of the data queue, which may hold any amount of data orsufficient data for an entire message payload or response data to beeasily held in one location in the queue, then a lock is obtained by theconsumer thread to exclude other processes or threads (e.g., theproducer thread) from accessing the queue of the send data stream atoperation 2311. The lock is briefly held at operation 2313, thisoperation updates the head position of the queue effectively discardingthe contents of the queue at the head position and releasing theposition in the queue or the memory associated with the data stream.This data can be part of the header or management data stored in thestate data of the data stream. At operation 2315, the lock can bereleased and the consumer process can continue to check for theavailable port and data to be written to the port at operation 2301.

FIG. 24 is a flowchart illustrating one embodiment of a producer methodfor shared data stream management in a virtual connection. The producerprocess is described in regard to the management of the data from thereceive data stream. However, one skilled in the art would understandthat the principles and operations of this process are applicable andadaptable to the management of the send data stream as well. For sake ofclarity the example of the management of a producer method as it isapplied to the send data stream is provided. The producer thread can beinstantiated at the time that the VCE is created and allotted a scheduleto service the virtual connection containing the data streams.

The operation 2401, the producer thread detects reception of dataassociated with a virtual connection between the client and the serverprocess that are communicating over a Fibre Channel network using SCSI.The incoming SCSI requests are analyzed to determine a virtualconnection that services the client messages at operation 2403. Atoperation 2405, a check is made whether the virtual connection exists.If the virtual connection does not exist then the virtual connection isinstantiated along with its data stream at operation 2407. If thevirtual connection and associated data stream already exists or if thevirtual connection has been instantiated, then a check is made whetherthe send data stream is full at operation 2409. If the data stream isfull, then the process at operation 2411 may have to wait until datastream space becomes available or provide notification of a lack of datastream space. If the data stream is full, then the SCSI request can berecorded as the pending operation for this virtual connection and heldin that state for a period of time indicated in the request. If therequest indicates a zero-valued timeout, or if the timeout expiresbefore any data becomes available, the SCSI request is completed with acompletion code indicating that no data was transferred. Data is notretrieved from an upstream source (e.g., a SCSI write/DATA_SENDoperation) until space is available in the data stream. For a DATA_SENDcase, the DATA_SEND operation includes a timeout value (e.g., onesecond). If the timeout expires and there is still no room in the datastream, then the DATA_SEND operation is completed with aNO_DATA_TRANSFERRED response. The client recognizes this response andarranges to retry transferring the data. More generally if anotification is returned to the client of the lack of space in the datastream, the client can function to throttle the data being sent by theclient, which slows down the rate of traffic to a manageable level.

At operation 2413, if the data stream is not full, then the receiveddata is written to the tail of the singly linked list (or similar queuestructure) of the data stream. The singly linked list is utilized forthis complex consumer/producer model to help ensure that the threads areminimally blocked by one another. A check is made after the write, todetermine whether the tail of the linked list is full at operation 2415.If the queue at the tail position is not full, then the processcontinues to receive data and write it to the tail of the queue withoutrequiring a lock.

However, if the tail of the queue is full, then the lock for the senddata stream is obtained at operation 2417. The position of the tail ofthe queue can then be updated at operation 2419 in the state of the datastream. After the update of the tail position has completed, then thelock is released allowing continued reading and writing to the send datastream by the consumer thread and producer thread of these data streamsfor the virtual connections in a given VCE.

FIG. 25 is a block diagram of one embodiment of a statistics managementmodule of a server Fibre Channel adapter. While the embodiments aredescribed in relation to statistics management for a server FibreChannel adapter, one skilled in the art would understand that theprinciple, processes and structures described herein below are providedby way of example and not limitations. The statistics managementprocesses and structures can be applied to other networks or computingdevices and combination thereof where monitoring is performed andmetrics collected. The diagram shows the components of a server FibreChannel adapter include a statistics management module 2501. The othercomponents of the server Fibre Channel adapter are described in furtherdetail herein above such as with regard to the example of FIG. 4A. Thestatistics management module 2501 generates and manages a set ofstatistics items 2503. Each statistics item tracks at least one metricrelated to the system such as the load on a VCE, load on a virtualconnection, endpoint throughput and similar metrics. The statisticsmanagement module 2501 and statistics items are designed to operatewithout requiring a locking mechanism. However, in alternate embodimentssuch a lock can be provided for each statistics item 2503. Operations onthe statistics items can be atomic, i.e., able to be completed withoutinterruption by other threads or events.

To manage the resource demand in terms of space and processing power,the statistics items 2503 can be structured to maintain a set of counterarrays or similar data structures that track the associated metric overdiffering time ranges. For example, the statistics items 2503 caninclude a creation timestamp when the statistic item was created and acurrent count showing a current value for the monitored metricrepresenting the total value over the time between creation and thecurrent time. For example, if the statistics item 2503 tracks throughputthe total count can be the total number of packets or bytes that havebeen transferred over the time of the statistic item 2503 existence orup to the last measurement. The statistics items 2503 can be updatedwith new measurements at regular intervals. To conserve space arrays ofthese regular intervals are maintained at varying levels of granularity.For example a first array, which could be referred to as a short term orrecent counter array, can contain measurements over a short time periodsuch as every 10 seconds. Other counter arrays can track measurementsover a larger time periods such as on a minute by minute basis, whichcan be referred to as a medium or long term counter array. Any number ofsuch arrays over any variation in granularity can be tracked. The arrayscan be structured to include timestamps (TS) indicating the timing ofthe recorded value paired with a recorded value for the given metric.

The statistics management module 2501 can service requests for data fora particular point in time or over a particular time interval. Typicallythis time interval is bounded at one end with a current time stamp andthe other end is bounded by a specific value of the request. Forexample, the request can be to get a metric from a statistics item 2503over the last 5 minutes. However, the statistics item 2503 may not havemeasured data that directly corresponds with this time period with oneend of the time period falling between measured data points. Theboundary metrics can be derived from the available data usinginterpolation to cure this defect of the counter arrays and measureddata set.

FIG. 26 is a flowchart illustrating one embodiment of a statisticalmonitoring process. The method shows a general process for determining aresponse to a request by a statistic management module. At operation2601, the process receives a request for a statistic value over adefined interval. The request can come from a load balancing process,VCE balancing process, or similar components of the server Fibre Channeladapter that make use of metrics that can be tracked by the statisticalmanagement module.

In response to the request, at operation the statistic management moduleaccesses the relevant statistics item and calculates a result value forthe requested statistic by adding together accrued values that arealready stored in the statistics item as measurements recorded in thedata arrays at the varying levels of granularity where these values fallwithin the received interval. These accrued values are added withinterpolated values that fall outside the defined interval along withvalues inside the interval using the available data arrays at thevarying levels of granularity. This provides greater accuracy where theolder bound of the received interval does not fall at the time of arecorded value including when it falls between the ranges of thedifferent data arrays of the statistical item. This can be accomplishedby using the last (oldest) recorded value in the interval and the first(newest) recorded value outside the interval, regardless of the dataarray each of these recorded values may be found in and interpolating aresult that matches the interval boundary.

FIG. 27 is a flowchart illustrating one embodiment of a statisticalmonitoring process having a set of specified cases for generatingmonitoring data for a given interval. In this example, the statisticsitem has two discrete arrays of recorded values from which to drawresults for statistics requests. The first data array is of values witha shorter interval (e.g., 10 seconds) or higher frequency, referred toin the illustration as a ‘short term’ data structure (e.g., an array).The second data array is of values with a medium interval (e.g., 1minute) or lower frequency, referred to in the illustration as a ‘mediumterm’ data structure (e.g., an array). The method is organized to handlevarious cases of where the start location for an interval of astatistical request falls relative to the short term and medium termdata structures. One skilled in the art would understand that the twoarray structure is provided by way of example and that the principlesand structures described herein can apply to any number of arrays havingany relative relationship in terms of ranges of coverage.

At operation 2701, the request for statistics over a specified intervalof time is received by the statistics management module. The request isanalyzed at operation 2703 to determine whether the start location ofthe specified interval falls in one of a set of defined cases. The startlocation as used herein indicates the earliest chronological boundary ofthe specified interval with the latest or most recent boundarycorresponding to the current timestamp.

In a first case, the start location falls between the current time stampand the most recent time stamp tracked in the short term data structure.At operation 2705, the result value in this case is calculated byinterpolation using the most recent value in the short term datastructure and a current value corresponding with the current time stamp,which is also tracked by the statistical item. The interpolated valuecan then be returned to the requestor at operation 2717.

In a second case, the start location falls within the time stamp rangeof the short term data structure. The request is analyzed at operation2707 by calculating the result value from recorded data in the shortterm data structure that falls before the start location (determined bytime stamp comparison) of the interval, that is the oldest value in theshort term data structure that falls within the specified interval. Thisvalue or all of the preceding values are added with an interpolatedvalue that is derived from the oldest value in the interval and the mostrecent value that is outside the interval. The resulting sum can then bereturned to the requestor at operation 2717.

In a third case, the start location falls between the short term andmedium term data structures, where there is no overlap between the timestamp ranges of these data structures. At operation 2709, the request isanalyzed by calculating the result value from the last value in theshort term data structure being added to an interpolated value derivedfrom the last (oldest) value in the short term data structure and themost recent value in the medium term data structure. The resulting sumcan then be returned to the requestor at operation 2717.

In a fourth case, the start location falls within the medium term datastructure and the short term data structure (specifically before thelast (oldest) value of the short term data structure), where there isoverlap between the two data structures. At operation 2711, the requestis analyzed by calculating the result value by interpolating a valuederived from a current value and a most recent (first) value of themedium term data structure. The resulting interpolated value can then bereturned to the requestor at operation 2717.

In a fifth case, the start location falls within the time stamp range ofthe medium term data structure. The request is analyzed at operation2713 by calculating the result value from recorded data in the mediumterm data structure that falls before the start location (determined bytime stamp comparison) of the interval, that is the oldest value in themedium term data structure that falls within the specified interval.This value or all of the preceding values are added with an interpolatedvalue that is derived from the oldest value in the interval and the mostrecent value that is outside the interval. The resulting sum can then bereturned to the requestor at operation 2717.

In a sixth case, the start location falls between the medium term datastructure and the creation time stamp. At operation 2715, the request isanalyzed by calculating the result value adding a last value or theoldest value in the medium term data structure with a result obtained byinterpolating a value derived from the value associated with thecreation time stamp and an oldest value of the medium term datastructure. The resulting sum can then be returned to the requestor atoperation 2717.

FIG. 28 is a block diagram of one embodiment of a VCE load balancingengine. In one embodiment, the resource pool is multi-tiered beingsimultaneously managed by a locality domain, a virtual connectionengine, and virtual connections. The same pool of resources can beassigned to a particular locality domain that in turn encompassesmultiple VCEs and each VCE can manage multiple virtual connections.Processes described herein above describe methods of distributingvirtual connections across VCEs. The present method spans rebalancingvirtual connections across VCEs and locality domains. The illustrationshows the relationship within the server Fibre Channel adapter 454 ofthe components of the locality domain 2801. Each locality domain 2801 istied to a discrete set of resources or a ‘resource pool.’ This resourcepool is then shared amongst the VCEs and virtual connections that areassigned to the locality domain 2801. During the operation of the serverFibre Channel adapter, the balance of the load or the distribution ofthe load across the set of locality domains can vary leading to a highload on one locality domain while other locality domains have low loads.A VCE load balancing engine 2811 can monitor the load distribution andrebalance the load across locality domains by reassigning virtualconnections to different VCEs or locality domains.

FIG. 29 is a flowchart illustrating one embodiment of a method of VCErebalancing. While the embodiments may be described in relation to adata backup system, this is provided by way of example and notlimitation. One skilled in the art would understand that the principles,structures and processes described in relation to this embodiment arealso applicable to other systems and functions. This method can beimplemented by a VCE load balancing engine 2811 or similar componentexecuted as part of a server Fibre Channel adapter on a server. In otherembodiments, the method is distributed across multiple products andcomponents. At operation 2901, the VCE load balancing engine can startthe VCE rebalancing process at a defined interval. The defined intervalcan have any length such that rebalancing is done on a regular basis atany reasonable frequency. The interval can be pre-programmed by aprogrammer or dynamically determined or locally inserted by a localuser. The method progresses through a set of possible rebalancingactions starting with a most preferred rebalancing option andprogressing to a least preferred rebalancing option.

At operation 2903, the VCE rebalancing module searches for a one-wayreassignment of a virtual connection from a busiest VCE and/or localitydomain to a least busy VCE and/or locality domain such that thereassignment places both the busiest VCE or locality domain and theleast busy VCE or locality domain into a target load range (i.e., arange of load values that are defined as acceptable load levels) withoutreversing the relative load order of the VCE or locality domainsinvolved in the reassignment. In one embodiment, when deciding whetherto move a virtual connection from one VCE or locality domain to another,the process can proceed in two stages first to prefer to move a virtualconnection from a busiest locality domain to a least busy localitydomain. More precisely, a virtual connection can be moved from a mostbusy VCE in the most busy locality domain to a least busy VCE in a leastbusy locality domain. Once, the locality domains are relativelybalanced, then the process can seek to move a virtual connection from amost busy VCE to a least busy VCE within a locality domain. Thistwo-level approach to rebalancing applies to each stage of the process.

The load order is the order from high to low or low to high of the loadof each VCE or locality domain. If a relative load order is maintainedafter an assignment, then the load of the busiest VCE or locality domainwill remain higher than the load of the least busy VCE or localitydomain after the reassignment. A one-way reassignment is a movement of aset of virtual connections from a respective VCE or locality domain toanother VCE or locality domain. In the one-way reassignment thereceiving VCE or locality domain keeps all other virtual connections orVCEs respectively.

If a one-way reassignment meeting this criteria is found at operation2905, then the one-way reassignment is schedule for execution atoperation 2907. The method then continues by waiting for the nextinterval or similarly proceeding to a subsequent iteration of theanalysis for rebalancing. In some embodiments, a single set ofreassignments are carried out with each iteration or at each interval.In other embodiments, multiple reassignments or iterations are carriedout at each interval. Relative load order can then be considered overall reassignments during a given interval or set of iterations. Thevirtual connection or VCE that is reassigned can have any loadassociated with it or the individual load of the virtual connection canbe unknown or inferred. In one example embodiment, the virtualconnection that is reassigned has a heavy load or the heaviest load. Inanother example embodiment, the virtual connection that is selected hasa load that, if reassigned, would place the VCE or locality domain ofits current assignment into an acceptable load range.

If a reassignment was not found, then at operation 2909 a search for atwo-way reassignment is carried out to find reassignments that move avirtual connection from the most busy VCE and/or locality domain to theleast busy VCE and/or locality domain and to move another virtualconnection from the least busy VCE and/or locality domain to the mostbusy VCE and/or locality domain. Thus, the VCEs or locality domains swapa set of virtual connections. The net effect of the swap is to reducethe load on the most busy VCE and/or locality domain and increase theload on the least busy VCE and/or locality domain. The two-wayreassignment provides a greater range of possible solutions, but comesat a higher expense in terms of computation and resources to identifythe reassignments and carry out the assignments. In one embodiment, therange of acceptable loads on each VCE or locality domain can be expandedand it can be allowed to reverse the relative load order of the VCEs andlocality domains. In other embodiments, any one or both of theserequirements may be waived to find a solution.

At operation 2911, if a search found a two-way reassignment, then thereassignment can be schedule for execution at operation 2907. Ifmultiple solutions are found at any stage, then tie-breakers can beutilized to select from the solutions, such as solutions that areclosest to a middle of the target load range or similar tie-breakingmetrics. The method then continues by waiting for the next interval orsimilarly proceeding to a subsequent iteration of the analysis forrebalancing. As discussed above, in other embodiments multipleiterations can be carried out at each interval.

At operation 2913, a search for a one-way reassignment of a virtualconnection is carried out to find a reassignment of a virtual connectionfrom a most busy VCE and/or locality domain to a least busy VCE and/orlocality domain. However, contrary to operation 2903, this one-wayreassignment is not required to result in the VCEs or locality domainsinvolved in the reassignment falling within a target load range, butthat still avoids reversing load order between the VCEs or localitydomains. If such a one-way reassignment is found at operation 2915, themethod proceeds to schedule the execution of the reassignment atoperation 2907.

Finally, if the previous searches do not reveal a reassignment thatmeets the established criteria, then, at operation 2917, the methodperforms a search for a one-way reassignment of a virtual connection tofind a reassignment of a virtual connection from a most busy VCE and/orlocality domain to a least busy VCE or locality domain. However,contrary to operations 2903 and 2913, this one-way reassignment is notrequired to result in the VCEs or locality domains involved in thereassignment falling within a target load range or to avoid reversingload order between the VCEs or locality domains. However, thereassignment is required to reduce the imbalance between the VCEs orlocality domains. That is, the difference in load of the source andtarget VCEs or locality domains must decrease after reassignment whencompared to current assignment. If such a one-way reassignment is found,then the method proceeds to schedule the execution of the reassignmentat operation 2907.

One skilled in the art would understand that this method of rebalancingis provided by way of example, rather than limitation. The method cansearch for or alter the criteria with logical permutations whileremaining consistent with the principles and structures described hereinabove. Such permutations can include greater use of two-wayreassignments or even reassignments involving one or more virtualconnections or more than two VCEs or locality domains.

FIG. 30 is a flowchart illustrating one embodiment of a method ofendpoint assignment. During the establishment of a virtual connection asdescribed herein above, a path between the client and the server FibreChannel adapter is chosen from a set of available paths along which theclient is able to detect the resource (e.g., a LUN) that it seeks toaccess. The server Fibre Channel adapter selects from amongst theseavailable paths and returns the selection to the client whenestablishing the virtual connection. Establishing this virtualconnection thus creates a load on the selected path including theendpoints of the path referred to as the initiator endpoint and targetendpoint where the target endpoint is tied to the resource at the serverand the initiator endpoint is tied to the process on the clientaccessing the resource. The initial path selection method attempts todistribute the load across the available paths and endpoints.

In the initial path selection method, at operation 3001, the serverFibre Channel adapter receives a set of available paths from the clientfor a connection request in the process of establishing the virtualconnection. The server Fibre Channel adapter can request or be providedwith load metrics for the target endpoints and the initiator endpointsfor each available path at operation 3003. The metrics can be requestedby the server Fibre Channel adapter from the statistics managementmodule or similar component of the server. The client can also providemetrics when requested or along with the connection request.

Using the load data, the server Fibre Channel adapter selects paths fromthe set of available paths that have the least busy target endpoints atoperation 3005. If there are multiple target endpoints having the samelow level of busyness, then a secondary consideration of the busyness ofthe initiator endpoint can be utilized to tie-break. In otherembodiment, the relative busyness of the target endpoint and theinitiator endpoint can be differently weighted. The busyness of anendpoint can be determined through any set of metrics. In one exampleembodiment, the metrics can include virtual connection count for eachendpoint, throughput, operations executed. These metrics can becollected at any interval such as at 10 second intervals or one minuteintervals. The collection of the metrics over time can also be analyzedto determine trends with the metric that can indicate whether theendpoint is becoming more or less busy. After selection using thismethod, the result is returned to the client to establish the connectionwith the server.

FIG. 31 is a flowchart illustrating one embodiment of a method ofendpoint rebalancing. The endpoint rebalancing is described in terms oftarget endpoint rebalancing. However, one skilled in the art wouldunderstand that the principles and operations described in regard totarget endpoint rebalancing can also be applied or adapted to initiatorendpoint rebalancing. The server Fibre Channel adapter is capable ofrequesting and managing the migration of a virtual connection from acurrent active path to an alternative path from the set of availablepaths. Overall load reduction can be achieved by identifying virtualconnections that are using more busy endpoints and requesting ormanaging the migration of these endpoints to alternative paths that areless busy.

An example method of virtual connection path migration and rebalancingis illustrated by way of example and not limitation. At operation 3101,the rebalancing process can be started at a defined interval. Therebalancing can be re-analyzed at any frequency using any intervalbetween iterations. At operation 3103, the method proceeds by receivingor retrieving monitored load data for the set of target endpoints. Thisdata can be obtained from the statistics management module or similarsources. The load can be calculated using any metric such as bytestransmitted using SCSI or similar metrics. If the metrics measuring theload on any endpoint show that the endpoint is below a definedthreshold, then the endpoint is disqualified at operation 3105. Thisdisqualification removes the endpoint as a candidate for load reductionssince the load is already sufficiently low on the endpoint.

The method continues at operation 3107, where the method loops over theset of remaining endpoints associated with available paths by selectinga next endpoint that is the most busy target endpoint. The iterationover the set of endpoints continues until the set of endpoints isexhausted. The selected target endpoint is marked as disqualified,removing it from consideration for further processing in lateriterations. The goal of each iteration is to attempt to identify a setof virtual connections that are currently using paths to the selectedtarget endpoint and that can be migrated to other alternative paths tothereby reduce the load on the target endpoint. In selecting whichvirtual connections to migrate, the method seeks to consider thecharacteristics of both the target endpoint and the initiator endpointof each alternative path. Amongst the set of possible solutions, it canbe preferred to migrate virtual connections to a path whose destinationtarget endpoint is not busy at all as compared to being less busyoverall. The method also prefers migrating virtual connections such thatthe imbalance between a source and destination of the migration of theendpoint is reduced, but the high/low relationship is still retained(i.e., the relative load order is not reversed.)

The method also prefers to migrate virtual connections to paths with thesame initiator endpoint, then second to a less-busy initiator, andfinally to a more busy initiator endpoint. To implement thesepreferences as well as other endpoint migration rules or suggestions,the method loops over a set of all virtual connections to identifyvirtual connections where the overall load is improved by the movementof the virtual connection to a new path. The preferences are applied bythe use of a set of categories that cover the preferences as well aspermutations of each preference. There can be any number of separatecategories defined and associated with each of the endpoints and paths.The names or identifiers of the categories can be descriptive, have anynumber and can be utilized divide the set of endpoints into groups tiedto the busyness of each endpoint, whether the use of the path couldcause a reversal of relative load order or similar criteria. In oneexample embodiment the categories can have an inherent order tied totheir preference as a category. For example, a category where there isan unbusy target endpoint, an unchanged load order, and the sameinitiator is utilized. Any number of different criteria can be used witheach additional criteria increasing the amount of possible categories.The classification of all alternate paths of all VCs assigned to aselected target endpoint according to the busyness of the paths andtheir endpoints along with load order and imbalance is performed atoperation 3111.

The set of categorized alternate paths can then be examined to identifythe path with the highest ordered categorization at operation 3113. Thismethod can continue to look for other alternate paths for other targetendpoints or virtual connections until the lower ordered categories arereached. These lower ordered categories can be skipped or bypassed if atarget level of load reduction is reached overall, saving thecomputational resources for carrying out these now unnecessarycomparisons. If the target level of load reduction is not reached, thenthe lower order classification can be examined and utilized. For eachdiscovered path the virtual connection and path index or similaridentifying information can be recorded at operation 3115. This set ofalternative paths can then be returned for implementation of themigration by the respective VCE or similar entity. As mentioned above,the process is generally applicable to both target and initiatorendpoint analysis.

Some portions of the preceding detailed descriptions have been presentedin terms of algorithms and symbolic representations of operations ondata bits within a computer memory. These algorithmic descriptions andrepresentations are the ways used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of operations leading to adesired result. The operations are those requiring physicalmanipulations of physical quantities.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the above discussion, itis appreciated that throughout the description, discussions utilizingterms such as those set forth in the claims below, refer to the actionand processes of a computer system, or similar electronic computingdevice, that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

Embodiments of the invention also relate to an apparatus for performingthe operations herein. Such a computer program is stored in anon-transitory computer readable medium. A machine-readable mediumincludes any mechanism for storing information in a form readable by amachine (e.g., a computer). For example, a machine-readable (e.g.,computer-readable) medium includes a machine (e.g., a computer) readablestorage medium (e.g., read only memory (“ROM”), random access memory(“RAM”), magnetic disk storage media, optical storage media, flashmemory devices).

The processes or methods depicted in the preceding figures can beperformed by processing logic that comprises hardware (e.g., circuitry,dedicated logic, etc.), software (e.g., embodied on a non-transitorycomputer readable medium), or a combination of both. Although theprocesses or methods are described above in terms of some sequentialoperations, it should be appreciated that some of the operationsdescribed can be performed in a different order. Moreover, someoperations can be performed in parallel rather than sequentially.

Embodiments of the present invention are not described with reference toany particular programming language. It will be appreciated that avariety of programming languages can be used to implement the teachingsof embodiments of the invention as described herein.

In the foregoing Specification, embodiments of the invention have beendescribed with reference to specific exemplary embodiments thereof. Itwill be evident that various modifications can be made thereto withoutdeparting from the broader spirit and scope of the invention as setforth in the following claims. The Specification and drawings are,accordingly, to be regarded in an illustrative sense rather than arestrictive sense.

What is claimed is:
 1. A method implemented by a server for loadbalancing of virtual connections over a fiber channel network, thevirtual connections using a Small Computer System Interface (SCSI), themethod comprising: identifying, by a fibre channel adapter executed by ahardware processor of the server, a set of locality domains, eachlocality domain comprising a hardware processor and memory, the fibrechannel adapter configured to assign at least one virtual connectionengine to each of the locality domains, wherein a virtual connectionengine comprises a hardware processor executing processing threads thatimplement a set of dynamically reassignable virtual connections;finding, by the fiber channel adapter, a one-way reassignment of avirtual connection (VC) from a busiest virtual connection engine (VCE)or locality domain within the server to a least busy VCE or localitydomain within the server that places the busiest VCE or locality domainand least busy VCE or locality domain in a target load range withoutreversing a load order; and executing the one-way reassignment by thefiber channel adapter to disassociate the VC from the busiest VCE orlocality domain and to reassociate the VC with the least busy VCE orlocality domain.
 2. The method of claim 1, further comprising: finding atwo-way reassignment of the VC from the busiest VCE or locality domainto the least busy VCE or locality domain and reassignment of another VCfrom the least busy VCE or locality domain to the busiest VCE orlocality domain in the target load range.
 3. The method of claim 1,further comprising: finding a one-way reassignment of the VC from thebusiest VCE or locality domain to the least busy VCE or locality domainthat places the busiest VCE or locality domain and the least busy VCE orlocality domain outside the target load range without reversing the loadorder.
 4. The method of claim 1, further comprising: finding a one-wayreassignment of the VC from the busiest VCE or locality domain to theleast busy VCE or locality domain that is outside the target load rangeand that reverses the load order.
 5. A server system implementing loadbalancing for virtual connections over a fibre channel network, thevirtual connections using a Small Computer System Interface (SCSI), theserver system comprising: a host fibre channel adapter to transmit andreceive data traffic over a fibre channel connection, the fibre channeladapter configured to identify a set of locality domains, each localitydomain comprising a hardware processor and memory, the fibre channeladaptor configured to assign at least one virtual connection engine toeach of the locality domains, wherein a virtual connection enginecomprises a hardware processor executing processing threads thatimplement a set of dynamically reassignable virtual connections; and aprocessor to execute the server fibre channel adapter, the server fibrechannel adapter includes a load balancing engine to manage load acrossthe set of locality domains and the virtual connection engines, the loadbalancing engine configured to find a one-way reassignment of a virtualconnection (VC) or from busiest virtual connection engine (VCE) orlocality domain within the server fibre channel adapter to least busyVCE or locality domain within the server fibre channel adapter thatplaces the busiest VCE or locality domain and least busy VCE or localitydomain in a target load range without reversing a load order, andexecute the reassignment.
 6. The server system of claim 5, wherein theload balancing engine is further configured to find a two-wayreassignment of the VC from the busiest VCE or locality domain to theleast busy VCE or locality domain and reassignment of another VC or VCEfrom the least busy VCE or locality domain to the busiest VCE orlocality domain in the target load range.
 7. The server system of claim5, wherein the load balancing engine is further configured to find aone-way reassignment of the VC from the busiest VCE or locality domainto the least busy VCE or locality domain that places the busiest VCE orlocality domain and the least busy VCE or locality domain outside thetarget load range without reversing the load order.
 8. The server systemof claim 5, wherein the load balancing engine is further configured tofind a one-way reassignment of the VC from the busiest VCE or localitydomain to the least busy VCE or locality domain that is outside thetarget load range and that reverses the load order.
 9. A non-transitorymachine readable medium having stored therein instructions to beexecuted by a server computer coupled to the machine readable medium,the instructions when executed by the server computer cause the servercomputer to: identify, by a fiber channel adapter executed by a hardwareprocessor of the server, a set of locality domains, each locality domaincomprising a hardware processor and memory, the fibre channel adapter isfurther configured to assign at least one virtual connection engine toeach of the locality domains, wherein a virtual connection enginecomprises a hardware processor executing processing threads thatimplement a set of dynamically reassignable virtual connections; find,by the fiber channel adapter, a one-way reassignment of a virtualconnection (VC) from a busiest virtual connection engine (VCE) orlocality domain within the server computer to a least busy VCE orlocality domain placing within the server computer that places thebusiest VCE or locality domain and the least busy VCE or locality domainin a target load range without reversing a load order; and execute theone-way reassignment by the fiber channel adapter to disassociate the VCfrom the busiest VCE or locality domain and to reassociate the VC withthe least busy VCE or locality domain.
 10. The non-transitory machinereadable medium of claim 9, wherein the instructions when executed bythe server computer further cause the server computer to: find a two-wayreassignment of the VC from the busiest VCE or locality domain to theleast busy VCE or locality domain and reassignment of another VC fromthe least busy VCE or locality domain to the busiest VCE or localitydomain in the target load range.
 11. The non-transitory machine readablemedium of claim 9, wherein the instructions when executed by the servercomputer further cause the server computer to: find a one-wayreassignment of the VC from the busiest VCE or locality domain to theleast busy VCE or locality domain that places the busiest VCE orlocality domain and the least busy VCE or locality domain outside thetarget load range without reversing the load order.
 12. Thenon-transitory machine readable medium of claim 9, wherein theinstructions when executed by the server computer further cause theserver computer to: find a one-way reassignment of the VC from thebusiest VCE or locality domain to the least busy VCE or locality domainthat is outside the target load range and that reverses the load order.